Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250112 02:52:19.821615 416 test_util.cc:274] Using random seed: 1788520460
2025-01-12T02:52:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-12T02:52:19Z Disabled control of system clock
I20250112 02:52:19.849917 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu
/tmp/dist-test-tasko6SHYo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42341
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42341,127.0.104.61:38771,127.0.104.60:39987
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250112 02:52:20.139698 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:52:20.139930 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:52:20.140022 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 02:52:20.146699 424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 02:52:20.146786 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:52:20.146821 424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 02:52:20.146860 424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 02:52:20.156744 424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42341,127.0.104.61:38771,127.0.104.60:39987
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4135
ASAN enabled
I20250112 02:52:20.158419 424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:52:20.160254 424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 02:52:20.170276 429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:52:20.171130 430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:52:20.171908 432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:52:20.172852 424 server_base.cc:1034] running on GCE node
I20250112 02:52:20.173493 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:52:20.174583 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:52:20.175765 424 hybrid_clock.cc:648] HybridClock initialized: now 1736650340175698 us; error 58 us; skew 500 ppm
I20250112 02:52:20.176023 424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:52:20.178314 424 webserver.cc:458] Webserver started at http://127.0.104.62:36515/ using document root <none> and password file <none>
I20250112 02:52:20.178951 424 fs_manager.cc:362] Metadata directory not provided
I20250112 02:52:20.179066 424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:52:20.179317 424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 02:52:20.181587 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-0/data/instance:
uuid: "436220c23fa948efb5b6ca15875d8b0e"
format_stamp: "Formatted at 2025-01-12 02:52:20 on dist-test-slave-h743"
I20250112 02:52:20.182260 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-0/wal/instance:
uuid: "436220c23fa948efb5b6ca15875d8b0e"
format_stamp: "Formatted at 2025-01-12 02:52:20 on dist-test-slave-h743"
I20250112 02:52:20.186518 424 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250112 02:52:20.189298 438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:52:20.190352 424 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250112 02:52:20.190479 424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-0/wal
uuid: "436220c23fa948efb5b6ca15875d8b0e"
format_stamp: "Formatted at 2025-01-12 02:52:20 on dist-test-slave-h743"
I20250112 02:52:20.190621 424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:52:20.207197 424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:52:20.207857 424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:52:20.208114 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:52:20.225183 424 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:42341
I20250112 02:52:20.225219 489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:42341 every 8 connection(s)
I20250112 02:52:20.226315 424 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-0/data/info.pb
I20250112 02:52:20.229557 490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:20.232193 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu as pid 424
I20250112 02:52:20.232458 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-0/wal/instance
I20250112 02:52:20.235265 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu
/tmp/dist-test-tasko6SHYo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:38771
--webserver_interface=127.0.104.61
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42341,127.0.104.61:38771,127.0.104.60:39987
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 02:52:20.236119 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } has no permanent_uuid. Determining permanent_uuid...
I20250112 02:52:20.250923 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } has no permanent_uuid. Determining permanent_uuid...
W20250112 02:52:20.251814 439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection refused (error 111)
W20250112 02:52:20.253032 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:38771: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection refused (error 111)
I20250112 02:52:20.302285 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } attempt: 1
W20250112 02:52:20.303820 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:38771: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection refused (error 111)
I20250112 02:52:20.372016 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } attempt: 2
W20250112 02:52:20.373680 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:38771: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection refused (error 111)
I20250112 02:52:20.465006 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } attempt: 3
W20250112 02:52:20.466634 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:38771: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection refused (error 111)
W20250112 02:52:20.531049 492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:52:20.531342 492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:52:20.531450 492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 02:52:20.538796 492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 02:52:20.538901 492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:52:20.539021 492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 02:52:20.539088 492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 02:52:20.547577 492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42341,127.0.104.61:38771,127.0.104.60:39987
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:38771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4135
ASAN enabled
I20250112 02:52:20.548990 492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:52:20.550421 492 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 02:52:20.559931 501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:52:20.560729 503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:52:20.561573 500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:52:20.562404 492 server_base.cc:1034] running on GCE node
I20250112 02:52:20.563001 492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:52:20.564139 492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:52:20.565326 492 hybrid_clock.cc:648] HybridClock initialized: now 1736650340565280 us; error 26 us; skew 500 ppm
I20250112 02:52:20.565659 492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:52:20.567562 492 webserver.cc:458] Webserver started at http://127.0.104.61:33823/ using document root <none> and password file <none>
I20250112 02:52:20.568168 492 fs_manager.cc:362] Metadata directory not provided
I20250112 02:52:20.568257 492 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:52:20.568482 492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 02:52:20.570719 492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/data/instance:
uuid: "f4a49a5756b143fcb767c3f41463f8d9"
format_stamp: "Formatted at 2025-01-12 02:52:20 on dist-test-slave-h743"
I20250112 02:52:20.571560 492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/wal/instance:
uuid: "f4a49a5756b143fcb767c3f41463f8d9"
format_stamp: "Formatted at 2025-01-12 02:52:20 on dist-test-slave-h743"
I20250112 02:52:20.575838 492 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20250112 02:52:20.578696 509 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:52:20.579885 492 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250112 02:52:20.580026 492 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/data,/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/wal
uuid: "f4a49a5756b143fcb767c3f41463f8d9"
format_stamp: "Formatted at 2025-01-12 02:52:20 on dist-test-slave-h743"
I20250112 02:52:20.580183 492 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:52:20.608970 492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:52:20.609622 492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:52:20.609863 492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:52:20.609838 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } attempt: 4
W20250112 02:52:20.611665 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:38771: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection refused (error 111)
I20250112 02:52:20.626747 492 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:38771
I20250112 02:52:20.626780 560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:38771 every 8 connection(s)
I20250112 02:52:20.627933 492 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/data/info.pb
I20250112 02:52:20.629755 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu as pid 492
I20250112 02:52:20.629925 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/wal/instance
I20250112 02:52:20.631263 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu
/tmp/dist-test-tasko6SHYo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:39987
--webserver_interface=127.0.104.60
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42341,127.0.104.61:38771,127.0.104.60:39987
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 02:52:20.631866 561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:20.638815 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } has no permanent_uuid. Determining permanent_uuid...
I20250112 02:52:20.650156 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } has no permanent_uuid. Determining permanent_uuid...
I20250112 02:52:20.658614 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } has no permanent_uuid. Determining permanent_uuid...
W20250112 02:52:20.659451 512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39987: connect: Connection refused (error 111)
W20250112 02:52:20.661293 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:39987: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39987: connect: Connection refused (error 111)
I20250112 02:52:20.710534 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } attempt: 1
W20250112 02:52:20.712193 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:39987: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39987: connect: Connection refused (error 111)
I20250112 02:52:20.780395 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } attempt: 2
W20250112 02:52:20.782078 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:39987: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39987: connect: Connection refused (error 111)
I20250112 02:52:20.873265 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } attempt: 3
W20250112 02:52:20.875085 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:39987: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39987: connect: Connection refused (error 111)
I20250112 02:52:20.910888 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } attempt: 5
I20250112 02:52:20.916673 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } has no permanent_uuid. Determining permanent_uuid...
W20250112 02:52:20.918203 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:39987: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39987: connect: Connection refused (error 111)
W20250112 02:52:20.935066 563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:52:20.935317 563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:52:20.935437 563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 02:52:20.942643 563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 02:52:20.942766 563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:52:20.942832 563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 02:52:20.942911 563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 02:52:20.951660 563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42341,127.0.104.61:38771,127.0.104.60:39987
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:39987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4135
ASAN enabled
I20250112 02:52:20.952953 563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:52:20.954414 563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 02:52:20.963801 571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:52:20.964577 572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:52:20.965368 574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:52:20.966017 563 server_base.cc:1034] running on GCE node
I20250112 02:52:20.966641 563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:52:20.967864 563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:52:20.969090 563 hybrid_clock.cc:648] HybridClock initialized: now 1736650340969049 us; error 51 us; skew 500 ppm
I20250112 02:52:20.969400 563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:52:20.969398 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } attempt: 1
W20250112 02:52:20.970942 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:39987: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39987: connect: Connection refused (error 111)
I20250112 02:52:20.971686 563 webserver.cc:458] Webserver started at http://127.0.104.60:42831/ using document root <none> and password file <none>
I20250112 02:52:20.972303 563 fs_manager.cc:362] Metadata directory not provided
I20250112 02:52:20.972395 563 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:52:20.972617 563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 02:52:20.974845 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/data/instance:
uuid: "dd25f03f25df49178a5c53c636ecf304"
format_stamp: "Formatted at 2025-01-12 02:52:20 on dist-test-slave-h743"
I20250112 02:52:20.975562 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/wal/instance:
uuid: "dd25f03f25df49178a5c53c636ecf304"
format_stamp: "Formatted at 2025-01-12 02:52:20 on dist-test-slave-h743"
I20250112 02:52:20.979997 563 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20250112 02:52:20.982765 580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:52:20.983904 563 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250112 02:52:20.984046 563 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/data,/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/wal
uuid: "dd25f03f25df49178a5c53c636ecf304"
format_stamp: "Formatted at 2025-01-12 02:52:20 on dist-test-slave-h743"
I20250112 02:52:20.984206 563 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:52:20.999764 563 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:52:21.000437 563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:52:21.000684 563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:52:21.018270 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } attempt: 4
I20250112 02:52:21.018942 563 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:39987
I20250112 02:52:21.018957 631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:39987 every 8 connection(s)
I20250112 02:52:21.020750 563 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/data/info.pb
I20250112 02:52:21.024896 633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:21.026446 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu as pid 563
I20250112 02:52:21.026626 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/wal/instance
I20250112 02:52:21.032363 633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } has no permanent_uuid. Determining permanent_uuid...
I20250112 02:52:21.037652 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Bootstrap starting.
I20250112 02:52:21.039173 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } attempt: 2
I20250112 02:52:21.041532 561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:21.042768 561 log.cc:826] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Log is configured to *not* fsync() on all Append() calls
I20250112 02:52:21.045830 633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } has no permanent_uuid. Determining permanent_uuid...
I20250112 02:52:21.047439 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: No bootstrap required, opened a new log
I20250112 02:52:21.057554 561 raft_consensus.cc:357] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:21.057808 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Bootstrap starting.
I20250112 02:52:21.058043 561 raft_consensus.cc:383] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:21.058233 561 raft_consensus.cc:738] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4a49a5756b143fcb767c3f41463f8d9, State: Initialized, Role: FOLLOWER
I20250112 02:52:21.058663 633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } has no permanent_uuid. Determining permanent_uuid...
I20250112 02:52:21.059161 561 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:21.060210 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:21.060564 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:52:21.061439 561 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 02:52:21.062093 490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:21.063318 490 log.cc:826] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Log is configured to *not* fsync() on all Append() calls
I20250112 02:52:21.069017 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: No bootstrap required, opened a new log
I20250112 02:52:21.074666 633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304: Bootstrap starting.
I20250112 02:52:21.077513 490 raft_consensus.cc:357] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:21.077872 490 raft_consensus.cc:383] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:21.078697 490 raft_consensus.cc:738] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436220c23fa948efb5b6ca15875d8b0e, State: Initialized, Role: FOLLOWER
I20250112 02:52:21.078775 633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:21.079595 490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:21.080170 633 log.cc:826] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304: Log is configured to *not* fsync() on all Append() calls
I20250112 02:52:21.080900 650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:21.081202 650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:52:21.081970 490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [sys.catalog]: configured and running, proceeding with master startup.
I20250112 02:52:21.086246 633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304: No bootstrap required, opened a new log
W20250112 02:52:21.089962 657 catalog_manager.cc:1559] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:52:21.090080 657 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:52:21.095656 633 raft_consensus.cc:357] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:21.095978 633 raft_consensus.cc:383] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:21.096123 633 raft_consensus.cc:738] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd25f03f25df49178a5c53c636ecf304, State: Initialized, Role: FOLLOWER
I20250112 02:52:21.096702 633 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:21.087385 492 master_runner.cc:186] Error getting master registration for 127.0.104.62:42341: OK, instance_id { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" instance_seqno: 1736650340220138 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250112 02:52:21.098840 492 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:52:21.099607 661 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:21.100176 661 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:52:21.100472 633 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [sys.catalog]: configured and running, proceeding with master startup.
W20250112 02:52:21.112399 515 tablet.cc:2367] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250112 02:52:21.121682 673 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:52:21.121861 673 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:52:21.114881 424 master_runner.cc:186] Error getting master registration for 127.0.104.60:39987: OK, instance_id { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" instance_seqno: 1736650341013682 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250112 02:52:21.126207 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:52:21.128037 641 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:21.128234 641 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:21.129666 641 leader_election.cc:290] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 436220c23fa948efb5b6ca15875d8b0e (127.0.104.62:42341), dd25f03f25df49178a5c53c636ecf304 (127.0.104.60:39987)
I20250112 02:52:21.130898 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4a49a5756b143fcb767c3f41463f8d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "436220c23fa948efb5b6ca15875d8b0e" is_pre_election: true
I20250112 02:52:21.130908 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4a49a5756b143fcb767c3f41463f8d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd25f03f25df49178a5c53c636ecf304" is_pre_election: true
I20250112 02:52:21.131362 607 raft_consensus.cc:2463] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4a49a5756b143fcb767c3f41463f8d9 in term 0.
I20250112 02:52:21.131358 465 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4a49a5756b143fcb767c3f41463f8d9 in term 0.
I20250112 02:52:21.131816 512 leader_election.cc:304] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd25f03f25df49178a5c53c636ecf304, f4a49a5756b143fcb767c3f41463f8d9; no voters:
I20250112 02:52:21.132174 641 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:21.132323 641 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:21.132416 641 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:21.135169 641 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:21.137003 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4a49a5756b143fcb767c3f41463f8d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "436220c23fa948efb5b6ca15875d8b0e"
I20250112 02:52:21.137182 465 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:21.141883 465 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4a49a5756b143fcb767c3f41463f8d9 in term 1.
I20250112 02:52:21.143972 512 leader_election.cc:304] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 436220c23fa948efb5b6ca15875d8b0e, f4a49a5756b143fcb767c3f41463f8d9; no voters:
I20250112 02:52:21.144515 641 leader_election.cc:290] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [CANDIDATE]: Term 1 election: Requested vote from peers 436220c23fa948efb5b6ca15875d8b0e (127.0.104.62:42341), dd25f03f25df49178a5c53c636ecf304 (127.0.104.60:39987)
I20250112 02:52:21.145064 641 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:21.145483 641 raft_consensus.cc:695] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 1 LEADER]: Becoming Leader. State: Replica: f4a49a5756b143fcb767c3f41463f8d9, State: Running, Role: LEADER
I20250112 02:52:21.146170 641 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:21.167044 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4a49a5756b143fcb767c3f41463f8d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd25f03f25df49178a5c53c636ecf304"
I20250112 02:52:21.167244 607 raft_consensus.cc:3054] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:21.171990 607 raft_consensus.cc:2463] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4a49a5756b143fcb767c3f41463f8d9 in term 1.
I20250112 02:52:21.179903 680 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4a49a5756b143fcb767c3f41463f8d9. Latest consensus state: current_term: 1 leader_uuid: "f4a49a5756b143fcb767c3f41463f8d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:21.180354 680 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: This master's current role is: LEADER
I20250112 02:52:21.184089 684 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
W20250112 02:52:21.190882 682 catalog_manager.cc:1559] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:52:21.190985 682 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:52:21.194581 684 catalog_manager.cc:1485] Initializing Kudu cluster ID...
W20250112 02:52:21.210256 444 tablet.cc:2367] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 02:52:21.212800 607 raft_consensus.cc:1270] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 1 FOLLOWER]: Refusing update from remote peer f4a49a5756b143fcb767c3f41463f8d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 02:52:21.213660 680 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:21.221560 465 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 1 FOLLOWER]: Refusing update from remote peer f4a49a5756b143fcb767c3f41463f8d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 02:52:21.222381 680 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:21.248445 650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4a49a5756b143fcb767c3f41463f8d9. Latest consensus state: current_term: 1 leader_uuid: "f4a49a5756b143fcb767c3f41463f8d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:21.248735 650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:52:21.251451 685 mvcc.cc:204] Tried to move back new op lower bound from 7113319797598146560 to 7113319797349486592. Current Snapshot: MvccSnapshot[applied={T|T < 7113319797598146560}]
W20250112 02:52:21.253221 586 tablet.cc:2367] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 02:52:21.258242 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f4a49a5756b143fcb767c3f41463f8d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:21.258527 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: This master's current role is: LEADER
I20250112 02:52:21.259035 687 mvcc.cc:204] Tried to move back new op lower bound from 7113319797598146560 to 7113319797349486592. Current Snapshot: MvccSnapshot[applied={T|T < 7113319797598146560}]
I20250112 02:52:21.260201 684 catalog_manager.cc:1348] Generated new cluster ID: 273ff2f1d16946fcac5345b54ec7471c
I20250112 02:52:21.260290 684 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 02:52:21.260192 661 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4a49a5756b143fcb767c3f41463f8d9. Latest consensus state: current_term: 1 leader_uuid: "f4a49a5756b143fcb767c3f41463f8d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:21.260429 661 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:52:21.272060 680 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f4a49a5756b143fcb767c3f41463f8d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:21.272329 680 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: This master's current role is: LEADER
I20250112 02:52:21.291868 686 mvcc.cc:204] Tried to move back new op lower bound from 7113319797598146560 to 7113319797349486592. Current Snapshot: MvccSnapshot[applied={T|T < 7113319797598146560}]
I20250112 02:52:21.298187 661 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f4a49a5756b143fcb767c3f41463f8d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:21.298394 661 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:52:21.310470 650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f4a49a5756b143fcb767c3f41463f8d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:21.310667 650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:52:21.316217 684 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 02:52:21.317729 684 catalog_manager.cc:1505] Loading token signing keys...
I20250112 02:52:21.338413 684 catalog_manager.cc:5899] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Generated new TSK 0
I20250112 02:52:21.340077 684 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 02:52:21.353976 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu
/tmp/dist-test-tasko6SHYo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42341,127.0.104.61:38771,127.0.104.60:39987
--builtin_ntp_servers=127.0.104.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20250112 02:52:21.777493 696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:52:21.777779 696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:52:21.777909 696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 02:52:21.785918 696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:52:21.786348 696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250112 02:52:21.795923 696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:42341,127.0.104.61:38771,127.0.104.60:39987
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4135
ASAN enabled
I20250112 02:52:21.797438 696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:52:21.799041 696 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 02:52:21.810739 701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:52:21.811565 702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:52:21.812124 704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:52:21.812794 696 server_base.cc:1034] running on GCE node
I20250112 02:52:21.813350 696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:52:21.814174 696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:52:21.815362 696 hybrid_clock.cc:648] HybridClock initialized: now 1736650341815295 us; error 45 us; skew 500 ppm
I20250112 02:52:21.815645 696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:52:21.821215 696 webserver.cc:458] Webserver started at http://127.0.104.1:34063/ using document root <none> and password file <none>
I20250112 02:52:21.821892 696 fs_manager.cc:362] Metadata directory not provided
I20250112 02:52:21.822000 696 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:52:21.822283 696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 02:52:21.824728 696 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/data/instance:
uuid: "289afc116aa74408aa5bff74e11c50d8"
format_stamp: "Formatted at 2025-01-12 02:52:21 on dist-test-slave-h743"
I20250112 02:52:21.825455 696 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/instance:
uuid: "289afc116aa74408aa5bff74e11c50d8"
format_stamp: "Formatted at 2025-01-12 02:52:21 on dist-test-slave-h743"
I20250112 02:52:21.830044 696 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250112 02:52:21.832942 710 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:52:21.834023 696 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:21.834165 696 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal
uuid: "289afc116aa74408aa5bff74e11c50d8"
format_stamp: "Formatted at 2025-01-12 02:52:21 on dist-test-slave-h743"
I20250112 02:52:21.834313 696 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:52:21.840281 711 raft_consensus.cc:491] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4a49a5756b143fcb767c3f41463f8d9)
I20250112 02:52:21.840588 711 raft_consensus.cc:513] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:21.841529 715 raft_consensus.cc:491] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f4a49a5756b143fcb767c3f41463f8d9)
I20250112 02:52:21.841706 715 raft_consensus.cc:513] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:21.842548 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd25f03f25df49178a5c53c636ecf304" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "436220c23fa948efb5b6ca15875d8b0e" is_pre_election: true
I20250112 02:52:21.842734 465 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd25f03f25df49178a5c53c636ecf304 in term 1.
I20250112 02:52:21.842963 715 leader_election.cc:290] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4a49a5756b143fcb767c3f41463f8d9 (127.0.104.61:38771), dd25f03f25df49178a5c53c636ecf304 (127.0.104.60:39987)
I20250112 02:52:21.842895 536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd25f03f25df49178a5c53c636ecf304" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f4a49a5756b143fcb767c3f41463f8d9" is_pre_election: true
I20250112 02:52:21.843225 711 leader_election.cc:290] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 436220c23fa948efb5b6ca15875d8b0e (127.0.104.62:42341), f4a49a5756b143fcb767c3f41463f8d9 (127.0.104.61:38771)
I20250112 02:52:21.843405 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "436220c23fa948efb5b6ca15875d8b0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd25f03f25df49178a5c53c636ecf304" is_pre_election: true
I20250112 02:52:21.843664 606 raft_consensus.cc:2463] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 436220c23fa948efb5b6ca15875d8b0e in term 1.
I20250112 02:52:21.843689 583 leader_election.cc:304] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 436220c23fa948efb5b6ca15875d8b0e, dd25f03f25df49178a5c53c636ecf304; no voters:
I20250112 02:52:21.843972 536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "436220c23fa948efb5b6ca15875d8b0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f4a49a5756b143fcb767c3f41463f8d9" is_pre_election: true
I20250112 02:52:21.844071 711 raft_consensus.cc:2798] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 02:52:21.844180 711 raft_consensus.cc:491] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f4a49a5756b143fcb767c3f41463f8d9)
I20250112 02:52:21.844285 711 raft_consensus.cc:3054] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 1 FOLLOWER]: Advancing to term 2
I20250112 02:52:21.844494 441 leader_election.cc:304] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 436220c23fa948efb5b6ca15875d8b0e, dd25f03f25df49178a5c53c636ecf304; no voters:
I20250112 02:52:21.844815 715 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 02:52:21.844962 715 raft_consensus.cc:491] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f4a49a5756b143fcb767c3f41463f8d9)
I20250112 02:52:21.845050 715 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 1 FOLLOWER]: Advancing to term 2
I20250112 02:52:21.846951 711 raft_consensus.cc:513] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:21.847612 711 leader_election.cc:290] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [CANDIDATE]: Term 2 election: Requested vote from peers 436220c23fa948efb5b6ca15875d8b0e (127.0.104.62:42341), f4a49a5756b143fcb767c3f41463f8d9 (127.0.104.61:38771)
I20250112 02:52:21.847693 715 raft_consensus.cc:513] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:21.847908 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd25f03f25df49178a5c53c636ecf304" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "436220c23fa948efb5b6ca15875d8b0e"
I20250112 02:52:21.848197 536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd25f03f25df49178a5c53c636ecf304" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f4a49a5756b143fcb767c3f41463f8d9"
I20250112 02:52:21.848361 715 leader_election.cc:290] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [CANDIDATE]: Term 2 election: Requested vote from peers f4a49a5756b143fcb767c3f41463f8d9 (127.0.104.61:38771), dd25f03f25df49178a5c53c636ecf304 (127.0.104.60:39987)
I20250112 02:52:21.848727 465 raft_consensus.cc:2388] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate dd25f03f25df49178a5c53c636ecf304 in current term 2: Already voted for candidate 436220c23fa948efb5b6ca15875d8b0e in this term.
I20250112 02:52:21.848670 536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "436220c23fa948efb5b6ca15875d8b0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f4a49a5756b143fcb767c3f41463f8d9"
I20250112 02:52:21.849192 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "436220c23fa948efb5b6ca15875d8b0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd25f03f25df49178a5c53c636ecf304"
I20250112 02:52:21.849253 583 leader_election.cc:304] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd25f03f25df49178a5c53c636ecf304; no voters: 436220c23fa948efb5b6ca15875d8b0e, f4a49a5756b143fcb767c3f41463f8d9
I20250112 02:52:21.849478 606 raft_consensus.cc:2388] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 436220c23fa948efb5b6ca15875d8b0e in current term 2: Already voted for candidate dd25f03f25df49178a5c53c636ecf304 in this term.
I20250112 02:52:21.849603 711 raft_consensus.cc:2743] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250112 02:52:21.849905 441 leader_election.cc:304] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 436220c23fa948efb5b6ca15875d8b0e; no voters: dd25f03f25df49178a5c53c636ecf304, f4a49a5756b143fcb767c3f41463f8d9
I20250112 02:52:21.850163 715 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250112 02:52:21.851436 696 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:52:21.852154 696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:52:21.852433 696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:52:21.853256 696 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 02:52:21.855086 696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 02:52:21.855154 696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:21.855265 696 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 02:52:21.855311 696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 02:52:21.886605 696 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:35007
I20250112 02:52:21.886644 824 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:35007 every 8 connection(s)
I20250112 02:52:21.887854 696 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/data/info.pb
I20250112 02:52:21.894374 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu as pid 696
I20250112 02:52:21.894600 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/instance
I20250112 02:52:21.902870 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu
/tmp/dist-test-tasko6SHYo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42341,127.0.104.61:38771,127.0.104.60:39987
--builtin_ntp_servers=127.0.104.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250112 02:52:21.925539 826 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42341
I20250112 02:52:21.925778 826 heartbeater.cc:463] Registering TS with master...
I20250112 02:52:21.926692 826 heartbeater.cc:510] Master 127.0.104.62:42341 requested a full tablet report, sending...
I20250112 02:52:21.928979 455 ts_manager.cc:194] Registered new tserver with Master: 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:21.931080 827 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:21.931205 827 heartbeater.cc:463] Registering TS with master...
I20250112 02:52:21.931442 465 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 2 FOLLOWER]: Rejecting Update request from peer f4a49a5756b143fcb767c3f41463f8d9 for earlier term 1. Current term is 2. Ops: []
I20250112 02:52:21.931594 827 heartbeater.cc:510] Master 127.0.104.61:38771 requested a full tablet report, sending...
I20250112 02:52:21.931962 825 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39987
I20250112 02:52:21.932047 825 heartbeater.cc:463] Registering TS with master...
I20250112 02:52:21.932277 606 raft_consensus.cc:1235] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 2 FOLLOWER]: Rejecting Update request from peer f4a49a5756b143fcb767c3f41463f8d9 for earlier term 1. Current term is 2. Ops: []
I20250112 02:52:21.932154 836 consensus_queue.cc:1046] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250112 02:52:21.932482 825 heartbeater.cc:510] Master 127.0.104.60:39987 requested a full tablet report, sending...
I20250112 02:52:21.932513 836 raft_consensus.cc:3049] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 1 LEADER]: Stepping down as leader of term 1
I20250112 02:52:21.933101 836 raft_consensus.cc:738] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f4a49a5756b143fcb767c3f41463f8d9, State: Running, Role: LEADER
I20250112 02:52:21.933338 837 consensus_queue.cc:1046] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250112 02:52:21.933684 836 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:21.933969 597 ts_manager.cc:194] Registered new tserver with Master: 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:21.934144 836 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 1 FOLLOWER]: Advancing to term 2
I20250112 02:52:21.937081 526 ts_manager.cc:194] Registered new tserver with Master: 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:22.092563 657 catalog_manager.cc:1603] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 02:52:22.124008 673 catalog_manager.cc:1260] Loaded cluster ID: 273ff2f1d16946fcac5345b54ec7471c
I20250112 02:52:22.124097 673 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: loading cluster ID for follower catalog manager: success
I20250112 02:52:22.126334 673 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: acquiring CA information for follower catalog manager: success
I20250112 02:52:22.128232 673 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 02:52:22.193264 682 catalog_manager.cc:1260] Loaded cluster ID: 273ff2f1d16946fcac5345b54ec7471c
I20250112 02:52:22.193375 682 catalog_manager.cc:1553] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304: loading cluster ID for follower catalog manager: success
I20250112 02:52:22.197132 682 catalog_manager.cc:1575] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304: acquiring CA information for follower catalog manager: success
I20250112 02:52:22.199329 682 catalog_manager.cc:1603] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250112 02:52:22.221864 835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:52:22.222072 835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:52:22.222162 835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 02:52:22.229876 835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:52:22.230072 835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250112 02:52:22.238377 835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:42341,127.0.104.61:38771,127.0.104.60:39987
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4135
ASAN enabled
I20250112 02:52:22.239869 835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:52:22.241297 835 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 02:52:22.252025 843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:52:22.252475 842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:52:22.253291 845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:52:22.253868 835 server_base.cc:1034] running on GCE node
I20250112 02:52:22.254352 835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:52:22.255084 835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:52:22.256285 835 hybrid_clock.cc:648] HybridClock initialized: now 1736650342256232 us; error 25 us; skew 500 ppm
I20250112 02:52:22.256538 835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:52:22.258383 835 webserver.cc:458] Webserver started at http://127.0.104.2:33907/ using document root <none> and password file <none>
I20250112 02:52:22.259008 835 fs_manager.cc:362] Metadata directory not provided
I20250112 02:52:22.259121 835 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:52:22.259389 835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 02:52:22.261634 835 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/data/instance:
uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
format_stamp: "Formatted at 2025-01-12 02:52:22 on dist-test-slave-h743"
I20250112 02:52:22.262308 835 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/instance:
uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
format_stamp: "Formatted at 2025-01-12 02:52:22 on dist-test-slave-h743"
I20250112 02:52:22.266748 835 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20250112 02:52:22.269721 851 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:52:22.270762 835 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:22.270893 835 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal
uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
format_stamp: "Formatted at 2025-01-12 02:52:22 on dist-test-slave-h743"
I20250112 02:52:22.271039 835 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:52:22.285322 835 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:52:22.285957 835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:52:22.286204 835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:52:22.286986 835 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 02:52:22.288297 835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 02:52:22.288370 835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 02:52:22.288487 835 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 02:52:22.288544 835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 02:52:22.319737 835 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:44451
I20250112 02:52:22.319777 963 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:44451 every 8 connection(s)
I20250112 02:52:22.320868 835 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/data/info.pb
I20250112 02:52:22.325387 836 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:22.325616 836 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:22.326385 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu as pid 835
I20250112 02:52:22.326478 836 leader_election.cc:290] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 436220c23fa948efb5b6ca15875d8b0e (127.0.104.62:42341), dd25f03f25df49178a5c53c636ecf304 (127.0.104.60:39987)
I20250112 02:52:22.326565 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/instance
I20250112 02:52:22.326926 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4a49a5756b143fcb767c3f41463f8d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd25f03f25df49178a5c53c636ecf304" is_pre_election: true
I20250112 02:52:22.327162 606 raft_consensus.cc:2463] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4a49a5756b143fcb767c3f41463f8d9 in term 2.
I20250112 02:52:22.327592 512 leader_election.cc:304] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd25f03f25df49178a5c53c636ecf304, f4a49a5756b143fcb767c3f41463f8d9; no voters:
I20250112 02:52:22.327879 836 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250112 02:52:22.327968 836 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:22.328066 836 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 2 FOLLOWER]: Advancing to term 3
I20250112 02:52:22.328986 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4a49a5756b143fcb767c3f41463f8d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "436220c23fa948efb5b6ca15875d8b0e" is_pre_election: true
I20250112 02:52:22.329210 465 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4a49a5756b143fcb767c3f41463f8d9 in term 2.
I20250112 02:52:22.330564 836 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:22.331115 836 leader_election.cc:290] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [CANDIDATE]: Term 3 election: Requested vote from peers 436220c23fa948efb5b6ca15875d8b0e (127.0.104.62:42341), dd25f03f25df49178a5c53c636ecf304 (127.0.104.60:39987)
I20250112 02:52:22.331657 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4a49a5756b143fcb767c3f41463f8d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "436220c23fa948efb5b6ca15875d8b0e"
I20250112 02:52:22.331846 465 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 2 FOLLOWER]: Advancing to term 3
I20250112 02:52:22.332000 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4a49a5756b143fcb767c3f41463f8d9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd25f03f25df49178a5c53c636ecf304"
I20250112 02:52:22.332226 606 raft_consensus.cc:3054] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 2 FOLLOWER]: Advancing to term 3
I20250112 02:52:22.333887 606 raft_consensus.cc:2463] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4a49a5756b143fcb767c3f41463f8d9 in term 3.
I20250112 02:52:22.334250 512 leader_election.cc:304] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd25f03f25df49178a5c53c636ecf304, f4a49a5756b143fcb767c3f41463f8d9; no voters:
I20250112 02:52:22.334328 465 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4a49a5756b143fcb767c3f41463f8d9 in term 3.
I20250112 02:52:22.334513 836 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 3 FOLLOWER]: Leader election won for term 3
I20250112 02:52:22.337488 836 raft_consensus.cc:695] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 3 LEADER]: Becoming Leader. State: Replica: f4a49a5756b143fcb767c3f41463f8d9, State: Running, Role: LEADER
I20250112 02:52:22.337747 836 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:22.339531 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39987
I20250112 02:52:22.339816 837 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4a49a5756b143fcb767c3f41463f8d9. Latest consensus state: current_term: 3 leader_uuid: "f4a49a5756b143fcb767c3f41463f8d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:22.340019 837 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: This master's current role is: LEADER
I20250112 02:52:22.340317 964 heartbeater.cc:463] Registering TS with master...
I20250112 02:52:22.341065 964 heartbeater.cc:510] Master 127.0.104.60:39987 requested a full tablet report, sending...
I20250112 02:52:22.341641 966 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:22.341758 966 heartbeater.cc:463] Registering TS with master...
I20250112 02:52:22.342226 966 heartbeater.cc:510] Master 127.0.104.61:38771 requested a full tablet report, sending...
I20250112 02:52:22.342427 597 ts_manager.cc:194] Registered new tserver with Master: fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:22.343118 526 ts_manager.cc:194] Registered new tserver with Master: fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:22.344563 972 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 02:52:22.346073 965 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42341
I20250112 02:52:22.346200 965 heartbeater.cc:463] Registering TS with master...
I20250112 02:52:22.346625 972 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 02:52:22.346683 965 heartbeater.cc:510] Master 127.0.104.62:42341 requested a full tablet report, sending...
I20250112 02:52:22.347612 972 catalog_manager.cc:1260] Loaded cluster ID: 273ff2f1d16946fcac5345b54ec7471c
I20250112 02:52:22.347685 972 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 02:52:22.348029 455 ts_manager.cc:194] Registered new tserver with Master: fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:22.349056 972 catalog_manager.cc:1505] Loading token signing keys...
I20250112 02:52:22.349939 972 catalog_manager.cc:5910] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Loaded TSK: 0
I20250112 02:52:22.350493 972 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 02:52:22.356554 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu
/tmp/dist-test-tasko6SHYo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42341,127.0.104.61:38771,127.0.104.60:39987
--builtin_ntp_servers=127.0.104.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250112 02:52:22.439932 526 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:48359
W20250112 02:52:22.662801 973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:52:22.663023 973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:52:22.663130 973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 02:52:22.669909 973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:52:22.670101 973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250112 02:52:22.679076 973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:42341,127.0.104.61:38771,127.0.104.60:39987
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4135
ASAN enabled
I20250112 02:52:22.680634 973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:52:22.682133 973 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 02:52:22.693671 981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:52:22.694510 979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:52:22.695093 978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:52:22.695878 973 server_base.cc:1034] running on GCE node
I20250112 02:52:22.696336 973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:52:22.697079 973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:52:22.698248 973 hybrid_clock.cc:648] HybridClock initialized: now 1736650342698187 us; error 42 us; skew 500 ppm
I20250112 02:52:22.698501 973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:52:22.700433 973 webserver.cc:458] Webserver started at http://127.0.104.3:46883/ using document root <none> and password file <none>
I20250112 02:52:22.701027 973 fs_manager.cc:362] Metadata directory not provided
I20250112 02:52:22.701148 973 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:52:22.701393 973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 02:52:22.704043 973 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/data/instance:
uuid: "f1e775296eb746db8b318f17d26820e6"
format_stamp: "Formatted at 2025-01-12 02:52:22 on dist-test-slave-h743"
I20250112 02:52:22.704677 973 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/instance:
uuid: "f1e775296eb746db8b318f17d26820e6"
format_stamp: "Formatted at 2025-01-12 02:52:22 on dist-test-slave-h743"
I20250112 02:52:22.708958 973 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250112 02:52:22.711671 987 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:52:22.712718 973 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250112 02:52:22.712845 973 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal
uuid: "f1e775296eb746db8b318f17d26820e6"
format_stamp: "Formatted at 2025-01-12 02:52:22 on dist-test-slave-h743"
I20250112 02:52:22.712980 973 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:52:22.753367 973 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:52:22.753995 973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:52:22.754225 973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:52:22.754961 973 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 02:52:22.756193 973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 02:52:22.756263 973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 02:52:22.756371 973 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 02:52:22.756419 973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 02:52:22.787214 973 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:44831
I20250112 02:52:22.787789 1099 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:44831 every 8 connection(s)
I20250112 02:52:22.788718 973 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/data/info.pb
I20250112 02:52:22.790516 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu as pid 973
I20250112 02:52:22.790686 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/instance
I20250112 02:52:22.804054 1102 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:22.804121 1101 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42341
I20250112 02:52:22.804339 1101 heartbeater.cc:463] Registering TS with master...
I20250112 02:52:22.804343 1102 heartbeater.cc:463] Registering TS with master...
I20250112 02:52:22.804965 1100 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39987
I20250112 02:52:22.805083 1100 heartbeater.cc:463] Registering TS with master...
I20250112 02:52:22.805748 1100 heartbeater.cc:510] Master 127.0.104.60:39987 requested a full tablet report, sending...
I20250112 02:52:22.805817 1101 heartbeater.cc:510] Master 127.0.104.62:42341 requested a full tablet report, sending...
I20250112 02:52:22.805840 1102 heartbeater.cc:510] Master 127.0.104.61:38771 requested a full tablet report, sending...
I20250112 02:52:22.806995 526 ts_manager.cc:194] Registered new tserver with Master: f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:22.807176 597 ts_manager.cc:194] Registered new tserver with Master: f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:22.807904 526 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:50403
I20250112 02:52:22.808718 455 ts_manager.cc:194] Registered new tserver with Master: f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:22.814858 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250112 02:52:22.839403 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_65d48d9effbb4fd1ab54eb9015244268"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:22.839425 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_8a8936070f0646c79225f150ad27442f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:22.840233 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_8db6a50cdd8a4fb9a4276fd240476bcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:22.841106 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65d48d9effbb4fd1ab54eb9015244268 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:22.841148 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a8936070f0646c79225f150ad27442f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:22.842489 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_3bec6710031c41f58551db6980c971cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:22.842993 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bec6710031c41f58551db6980c971cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:22.843991 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_d67df1b9735a458796c1beb5fd0e0f37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:22.844324 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_423308c5a06142d8ae48d9027e544490"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:22.844790 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d67df1b9735a458796c1beb5fd0e0f37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:22.844897 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_423308c5a06142d8ae48d9027e544490 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:22.846628 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_4b4e336cd8f6428eae1876e48286b64b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:22.847225 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b4e336cd8f6428eae1876e48286b64b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:22.848698 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_eedb329466a44101ad242ab1105c2462"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:22.849411 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eedb329466a44101ad242ab1105c2462 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:22.851954 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_1ae04ce4866246fc9e5fe0be72adcf64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:22.852325 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8db6a50cdd8a4fb9a4276fd240476bcd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:22.853724 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ae04ce4866246fc9e5fe0be72adcf64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:22.855453 517 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:49489
I20250112 02:52:22.856021 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_2fb8fbe4fc5c46a48ce9e1764fe2191d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:22.856451 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fb8fbe4fc5c46a48ce9e1764fe2191d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:22.858234 606 raft_consensus.cc:1270] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 3 FOLLOWER]: Refusing update from remote peer f4a49a5756b143fcb767c3f41463f8d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250112 02:52:22.859663 1130 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250112 02:52:22.869011 465 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 3 FOLLOWER]: Refusing update from remote peer f4a49a5756b143fcb767c3f41463f8d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250112 02:52:22.869710 1130 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250112 02:52:22.894413 1143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4a49a5756b143fcb767c3f41463f8d9. Latest consensus state: current_term: 3 leader_uuid: "f4a49a5756b143fcb767c3f41463f8d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:22.894768 1143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:52:22.917300 1143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f4a49a5756b143fcb767c3f41463f8d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:22.917565 1143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:52:22.933650 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_8a8936070f0646c79225f150ad27442f" } new_table_name: "default.Table_38c977627680467690f8534465803113" modify_external_catalogs: true
I20250112 02:52:22.934180 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_423308c5a06142d8ae48d9027e544490" } new_table_name: "default.Table_2275071ba627422c802e5b2506d5594c" modify_external_catalogs: true
I20250112 02:52:22.934345 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_4b4e336cd8f6428eae1876e48286b64b" } new_table_name: "default.Table_a7f9e8004af1434d9cef01112b8d1849" modify_external_catalogs: true
I20250112 02:52:22.934742 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_65d48d9effbb4fd1ab54eb9015244268" } modify_external_catalogs: true
I20250112 02:52:22.934893 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_65d48d9effbb4fd1ab54eb9015244268" } modify_external_catalogs: true
I20250112 02:52:22.935230 526 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_8db6a50cdd8a4fb9a4276fd240476bcd" } modify_external_catalogs: true
I20250112 02:52:22.935366 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_d67df1b9735a458796c1beb5fd0e0f37" } new_table_name: "default.Table_199725d90df746acb9c1beb33de2ff4b" modify_external_catalogs: true
I20250112 02:52:22.935570 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_3bec6710031c41f58551db6980c971cb" } new_table_name: "default.Table_77d3bedb6a0c45b1a6b23ed2df9f4593" modify_external_catalogs: true
I20250112 02:52:22.936094 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_1ae04ce4866246fc9e5fe0be72adcf64" } modify_external_catalogs: true
I20250112 02:52:22.937114 1127 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f4a49a5756b143fcb767c3f41463f8d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:22.937320 1127 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: This master's current role is: LEADER
I20250112 02:52:22.941111 526 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_8db6a50cdd8a4fb9a4276fd240476bcd" } modify_external_catalogs: true
I20250112 02:52:22.941192 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_1ae04ce4866246fc9e5fe0be72adcf64" } modify_external_catalogs: true
I20250112 02:52:22.941817 827 heartbeater.cc:502] Master 127.0.104.61:38771 was elected leader, sending a full tablet report...
I20250112 02:52:22.953269 1140 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4a49a5756b143fcb767c3f41463f8d9. Latest consensus state: current_term: 3 leader_uuid: "f4a49a5756b143fcb767c3f41463f8d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:22.953596 1140 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [sys.catalog]: This master's current role is: FOLLOWER
W20250112 02:52:22.953773 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 600249dc30c842ae9ab872f01ff4b6c5 has no consensus state
I20250112 02:52:22.955036 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu pid 563
I20250112 02:52:22.955379 1130 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f4a49a5756b143fcb767c3f41463f8d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:22.955617 1130 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: This master's current role is: LEADER
I20250112 02:52:22.956225 627 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 02:52:22.957397 521 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_2fb8fbe4fc5c46a48ce9e1764fe2191d" } modify_external_catalogs: true
I20250112 02:52:22.957549 521 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_2fb8fbe4fc5c46a48ce9e1764fe2191d" } modify_external_catalogs: true
I20250112 02:52:22.958212 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_eedb329466a44101ad242ab1105c2462" } modify_external_catalogs: true
I20250112 02:52:22.958456 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_eedb329466a44101ad242ab1105c2462" } modify_external_catalogs: true
W20250112 02:52:22.961858 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3092eb25d1ac4333a9e5323f67738998 has no consensus state
W20250112 02:52:22.962162 525 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3cff5dd0596c4809ad4b50b27729b48b has no consensus state
W20250112 02:52:22.962364 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ed69c15f3d1448b929c3e9c768deb1f has no consensus state
W20250112 02:52:22.962509 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8cef1a66738c4bb3bad7019ce64f2ec2 has no consensus state
I20250112 02:52:22.965232 521 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet f23b81e438364492949818fb659e709f
I20250112 02:52:22.983651 899 tablet_service.cc:1467] Processing CreateTablet for tablet 2bca7a6cecdc49aeb9aa4ae65beb7757 (DEFAULT_TABLE table=default.Table_8db6a50cdd8a4fb9a4276fd240476bcd [id=01845880c50e41378a0c744d3b67cdbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:22.984972 895 tablet_service.cc:1467] Processing CreateTablet for tablet e56a00ec3f5040b982c555d65f07d34d (DEFAULT_TABLE table=default.Table_1ae04ce4866246fc9e5fe0be72adcf64 [id=aa8325dff47c4d95832e0107cd5a17e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:22.985692 896 tablet_service.cc:1467] Processing CreateTablet for tablet 3092eb25d1ac4333a9e5323f67738998 (DEFAULT_TABLE table=default.Table_a7f9e8004af1434d9cef01112b8d1849 [id=bf2df21352c64c069c2a06c878911016]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:22.988243 894 tablet_service.cc:1467] Processing CreateTablet for tablet 600249dc30c842ae9ab872f01ff4b6c5 (DEFAULT_TABLE table=default.Table_2275071ba627422c802e5b2506d5594c [id=4abf3b98488543d2933165c2f5a11a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:22.989121 893 tablet_service.cc:1467] Processing CreateTablet for tablet aa767148c9a94a0582de4a77052e0867 (DEFAULT_TABLE table=default.Table_65d48d9effbb4fd1ab54eb9015244268 [id=3eb02559a95b4223bf8050df99296f16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:22.992090 897 tablet_service.cc:1467] Processing CreateTablet for tablet 8cef1a66738c4bb3bad7019ce64f2ec2 (DEFAULT_TABLE table=default.Table_77d3bedb6a0c45b1a6b23ed2df9f4593 [id=60486b9e81254d74b8e08a1cadde13bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:22.993116 898 tablet_service.cc:1467] Processing CreateTablet for tablet 3cff5dd0596c4809ad4b50b27729b48b (DEFAULT_TABLE table=default.Table_38c977627680467690f8534465803113 [id=05e179e7b80a40eaa0e3d425e84dd7a3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:22.997521 513 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 600249dc30c842ae9ab872f01ff4b6c5 (table default.Table_2275071ba627422c802e5b2506d5594c [id=4abf3b98488543d2933165c2f5a11a94], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 600249dc30c842ae9ab872f01ff4b6c5 consensus state has no leader
I20250112 02:52:22.997718 519 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 6ae269d4f0ae420b9b027b91c9c07488
I20250112 02:52:23.000371 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 600249dc30c842ae9ab872f01ff4b6c5 (DEFAULT_TABLE table=default.Table_2275071ba627422c802e5b2506d5594c [id=4abf3b98488543d2933165c2f5a11a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.000696 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed69c15f3d1448b929c3e9c768deb1f (DEFAULT_TABLE table=default.Table_199725d90df746acb9c1beb33de2ff4b [id=f20d22fe09e94ecb8dbe6d4f0a4a8b28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.000964 1031 tablet_service.cc:1467] Processing CreateTablet for tablet e56a00ec3f5040b982c555d65f07d34d (DEFAULT_TABLE table=default.Table_1ae04ce4866246fc9e5fe0be72adcf64 [id=aa8325dff47c4d95832e0107cd5a17e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.001626 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 600249dc30c842ae9ab872f01ff4b6c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.001729 1029 tablet_service.cc:1467] Processing CreateTablet for tablet aa767148c9a94a0582de4a77052e0867 (DEFAULT_TABLE table=default.Table_65d48d9effbb4fd1ab54eb9015244268 [id=3eb02559a95b4223bf8050df99296f16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.002154 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa767148c9a94a0582de4a77052e0867. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.003381 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ed69c15f3d1448b929c3e9c768deb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.005201 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 3092eb25d1ac4333a9e5323f67738998 (DEFAULT_TABLE table=default.Table_a7f9e8004af1434d9cef01112b8d1849 [id=bf2df21352c64c069c2a06c878911016]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.005649 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3092eb25d1ac4333a9e5323f67738998. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.005795 526 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 2bca7a6cecdc49aeb9aa4ae65beb7757
I20250112 02:52:23.006258 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet e56a00ec3f5040b982c555d65f07d34d
I20250112 02:52:23.000509 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 8cef1a66738c4bb3bad7019ce64f2ec2 (DEFAULT_TABLE table=default.Table_77d3bedb6a0c45b1a6b23ed2df9f4593 [id=60486b9e81254d74b8e08a1cadde13bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.007000 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet aa767148c9a94a0582de4a77052e0867
I20250112 02:52:23.007267 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cef1a66738c4bb3bad7019ce64f2ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.007890 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e56a00ec3f5040b982c555d65f07d34d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.000129 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 3cff5dd0596c4809ad4b50b27729b48b (DEFAULT_TABLE table=default.Table_38c977627680467690f8534465803113 [id=05e179e7b80a40eaa0e3d425e84dd7a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.007319 751 tablet_service.cc:1514] Processing DeleteTablet for tablet e56a00ec3f5040b982c555d65f07d34d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:22 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.008705 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cff5dd0596c4809ad4b50b27729b48b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.009021 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bca7a6cecdc49aeb9aa4ae65beb7757 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:22 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.008606 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bca7a6cecdc49aeb9aa4ae65beb7757 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:22 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.009955 760 tablet_service.cc:1467] Processing CreateTablet for tablet 2bca7a6cecdc49aeb9aa4ae65beb7757 (DEFAULT_TABLE table=default.Table_8db6a50cdd8a4fb9a4276fd240476bcd [id=01845880c50e41378a0c744d3b67cdbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.009397 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet e56a00ec3f5040b982c555d65f07d34d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:22 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.009517 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet aa767148c9a94a0582de4a77052e0867 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:22 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.011135 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bca7a6cecdc49aeb9aa4ae65beb7757. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:23.012550 512 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 2bca7a6cecdc49aeb9aa4ae65beb7757 because the tablet was not found. No further retry: Not found: Tablet not found: 2bca7a6cecdc49aeb9aa4ae65beb7757
W20250112 02:52:23.013242 512 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet e56a00ec3f5040b982c555d65f07d34d because the tablet was not found. No further retry: Not found: Tablet not found: e56a00ec3f5040b982c555d65f07d34d
W20250112 02:52:23.013744 512 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet aa767148c9a94a0582de4a77052e0867 because the tablet was not found. No further retry: Not found: Tablet not found: aa767148c9a94a0582de4a77052e0867
I20250112 02:52:23.015594 754 tablet_service.cc:1467] Processing CreateTablet for tablet aa767148c9a94a0582de4a77052e0867 (DEFAULT_TABLE table=default.Table_65d48d9effbb4fd1ab54eb9015244268 [id=3eb02559a95b4223bf8050df99296f16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.016000 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa767148c9a94a0582de4a77052e0867. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.016811 753 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed69c15f3d1448b929c3e9c768deb1f (DEFAULT_TABLE table=default.Table_199725d90df746acb9c1beb33de2ff4b [id=f20d22fe09e94ecb8dbe6d4f0a4a8b28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.017568 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ed69c15f3d1448b929c3e9c768deb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.019573 757 tablet_service.cc:1467] Processing CreateTablet for tablet 3092eb25d1ac4333a9e5323f67738998 (DEFAULT_TABLE table=default.Table_a7f9e8004af1434d9cef01112b8d1849 [id=bf2df21352c64c069c2a06c878911016]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.020045 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3092eb25d1ac4333a9e5323f67738998. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.022548 1173 tablet_bootstrap.cc:492] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:23.022884 756 tablet_service.cc:1467] Processing CreateTablet for tablet e56a00ec3f5040b982c555d65f07d34d (DEFAULT_TABLE table=default.Table_1ae04ce4866246fc9e5fe0be72adcf64 [id=aa8325dff47c4d95832e0107cd5a17e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.023723 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e56a00ec3f5040b982c555d65f07d34d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.026576 1173 tablet_bootstrap.cc:654] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.026487 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 2bca7a6cecdc49aeb9aa4ae65beb7757 (DEFAULT_TABLE table=default.Table_8db6a50cdd8a4fb9a4276fd240476bcd [id=01845880c50e41378a0c744d3b67cdbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.026989 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bca7a6cecdc49aeb9aa4ae65beb7757. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.028128 1173 log.cc:826] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6: Log is configured to *not* fsync() on all Append() calls
I20250112 02:52:23.033054 892 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed69c15f3d1448b929c3e9c768deb1f (DEFAULT_TABLE table=default.Table_199725d90df746acb9c1beb33de2ff4b [id=f20d22fe09e94ecb8dbe6d4f0a4a8b28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.033490 1173 tablet_bootstrap.cc:492] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:23.033752 1173 ts_tablet_manager.cc:1397] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20250112 02:52:23.038166 520 master_service.cc:946] ReplaceTablet: received request to replace tablet aa767148c9a94a0582de4a77052e0867 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:23.042220 520 master_service.cc:946] ReplaceTablet: received request to replace tablet 600249dc30c842ae9ab872f01ff4b6c5 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:23.043762 1173 raft_consensus.cc:357] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.044178 1173 raft_consensus.cc:383] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.044266 1173 raft_consensus.cc:738] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.047811 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bca7a6cecdc49aeb9aa4ae65beb7757. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.047194 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bca7a6cecdc49aeb9aa4ae65beb7757 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:22 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.049017 889 tablet_service.cc:1514] Processing DeleteTablet for tablet aa767148c9a94a0582de4a77052e0867 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:22 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.049921 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 600249dc30c842ae9ab872f01ff4b6c5
I20250112 02:52:23.050369 520 catalog_manager.cc:6596] ReplaceTablet: tablet 600249dc30c842ae9ab872f01ff4b6c5 deleted and replaced by tablet d9125dd3f9544a54ac373ae0f90cce5f
I20250112 02:52:23.044996 1173 consensus_queue.cc:260] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.050947 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 600249dc30c842ae9ab872f01ff4b6c5 with delete_type TABLET_DATA_DELETED (replaced by tablet d9125dd3f9544a54ac373ae0f90cce5f) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.051090 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 600249dc30c842ae9ab872f01ff4b6c5 with delete_type TABLET_DATA_DELETED (replaced by tablet d9125dd3f9544a54ac373ae0f90cce5f) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.051446 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e56a00ec3f5040b982c555d65f07d34d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:23.051713 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 600249dc30c842ae9ab872f01ff4b6c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 600249dc30c842ae9ab872f01ff4b6c5 already in progress: creating tablet
I20250112 02:52:23.052227 890 tablet_service.cc:1514] Processing DeleteTablet for tablet e56a00ec3f5040b982c555d65f07d34d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:22 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.026214 758 tablet_service.cc:1467] Processing CreateTablet for tablet 8cef1a66738c4bb3bad7019ce64f2ec2 (DEFAULT_TABLE table=default.Table_77d3bedb6a0c45b1a6b23ed2df9f4593 [id=60486b9e81254d74b8e08a1cadde13bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.054553 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cef1a66738c4bb3bad7019ce64f2ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.059266 890 tablet_service.cc:1467] Processing CreateTablet for tablet d9125dd3f9544a54ac373ae0f90cce5f (DEFAULT_TABLE table=default.Table_2275071ba627422c802e5b2506d5594c [id=4abf3b98488543d2933165c2f5a11a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.060223 1029 tablet_service.cc:1467] Processing CreateTablet for tablet d9125dd3f9544a54ac373ae0f90cce5f (DEFAULT_TABLE table=default.Table_2275071ba627422c802e5b2506d5594c [id=4abf3b98488543d2933165c2f5a11a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.060586 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9125dd3f9544a54ac373ae0f90cce5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.061430 755 tablet_service.cc:1467] Processing CreateTablet for tablet 600249dc30c842ae9ab872f01ff4b6c5 (DEFAULT_TABLE table=default.Table_2275071ba627422c802e5b2506d5594c [id=4abf3b98488543d2933165c2f5a11a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.068574 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 600249dc30c842ae9ab872f01ff4b6c5 with delete_type TABLET_DATA_DELETED (replaced by tablet d9125dd3f9544a54ac373ae0f90cce5f) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.069124 520 master_service.cc:946] ReplaceTablet: received request to replace tablet 3092eb25d1ac4333a9e5323f67738998 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:23.069574 752 tablet_service.cc:1514] Processing DeleteTablet for tablet aa767148c9a94a0582de4a77052e0867 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:22 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.067744 759 tablet_service.cc:1467] Processing CreateTablet for tablet 3cff5dd0596c4809ad4b50b27729b48b (DEFAULT_TABLE table=default.Table_38c977627680467690f8534465803113 [id=05e179e7b80a40eaa0e3d425e84dd7a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.068383 749 tablet_service.cc:1467] Processing CreateTablet for tablet d9125dd3f9544a54ac373ae0f90cce5f (DEFAULT_TABLE table=default.Table_2275071ba627422c802e5b2506d5594c [id=4abf3b98488543d2933165c2f5a11a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.073714 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 600249dc30c842ae9ab872f01ff4b6c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.075205 526 master_service.cc:946] ReplaceTablet: received request to replace tablet 3092eb25d1ac4333a9e5323f67738998 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:23.077014 522 master_service.cc:946] ReplaceTablet: received request to replace tablet d9125dd3f9544a54ac373ae0f90cce5f from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:23.077193 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 3092eb25d1ac4333a9e5323f67738998
I20250112 02:52:23.077756 520 catalog_manager.cc:6596] ReplaceTablet: tablet 3092eb25d1ac4333a9e5323f67738998 deleted and replaced by tablet 9c9a803ff19645328107d3b2fabb90c6
I20250112 02:52:23.078729 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 3092eb25d1ac4333a9e5323f67738998 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c9a803ff19645328107d3b2fabb90c6) from {username='slave'} at 127.0.0.1:50844
W20250112 02:52:23.079250 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 3092eb25d1ac4333a9e5323f67738998 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3092eb25d1ac4333a9e5323f67738998 already in progress: creating tablet
I20250112 02:52:23.082341 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cff5dd0596c4809ad4b50b27729b48b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.082932 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet d9125dd3f9544a54ac373ae0f90cce5f
I20250112 02:52:23.083369 522 catalog_manager.cc:6596] ReplaceTablet: tablet d9125dd3f9544a54ac373ae0f90cce5f deleted and replaced by tablet d02caa04543d43e086f5cd56ac1ead1a
I20250112 02:52:23.083885 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet d9125dd3f9544a54ac373ae0f90cce5f with delete_type TABLET_DATA_DELETED (replaced by tablet d02caa04543d43e086f5cd56ac1ead1a) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.084033 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 3092eb25d1ac4333a9e5323f67738998 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c9a803ff19645328107d3b2fabb90c6) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:23.084372 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet d9125dd3f9544a54ac373ae0f90cce5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9125dd3f9544a54ac373ae0f90cce5f already in progress: creating tablet
I20250112 02:52:23.086195 889 tablet_service.cc:1514] Processing DeleteTablet for tablet d9125dd3f9544a54ac373ae0f90cce5f with delete_type TABLET_DATA_DELETED (replaced by tablet d02caa04543d43e086f5cd56ac1ead1a) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.089270 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 3092eb25d1ac4333a9e5323f67738998 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c9a803ff19645328107d3b2fabb90c6) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.090054 889 tablet_service.cc:1467] Processing CreateTablet for tablet 9c9a803ff19645328107d3b2fabb90c6 (DEFAULT_TABLE table=default.Table_a7f9e8004af1434d9cef01112b8d1849 [id=bf2df21352c64c069c2a06c878911016]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.092160 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 9c9a803ff19645328107d3b2fabb90c6 (DEFAULT_TABLE table=default.Table_a7f9e8004af1434d9cef01112b8d1849 [id=bf2df21352c64c069c2a06c878911016]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.092527 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c9a803ff19645328107d3b2fabb90c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.096740 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9125dd3f9544a54ac373ae0f90cce5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.101660 888 tablet_service.cc:1467] Processing CreateTablet for tablet d02caa04543d43e086f5cd56ac1ead1a (DEFAULT_TABLE table=default.Table_2275071ba627422c802e5b2506d5594c [id=4abf3b98488543d2933165c2f5a11a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.102093 1030 tablet_service.cc:1467] Processing CreateTablet for tablet d02caa04543d43e086f5cd56ac1ead1a (DEFAULT_TABLE table=default.Table_2275071ba627422c802e5b2506d5594c [id=4abf3b98488543d2933165c2f5a11a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.102463 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d02caa04543d43e086f5cd56ac1ead1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.101819 750 tablet_service.cc:1467] Processing CreateTablet for tablet 9c9a803ff19645328107d3b2fabb90c6 (DEFAULT_TABLE table=default.Table_a7f9e8004af1434d9cef01112b8d1849 [id=bf2df21352c64c069c2a06c878911016]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.102886 752 tablet_service.cc:1514] Processing DeleteTablet for tablet d9125dd3f9544a54ac373ae0f90cce5f with delete_type TABLET_DATA_DELETED (replaced by tablet d02caa04543d43e086f5cd56ac1ead1a) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.104646 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c9a803ff19645328107d3b2fabb90c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.105856 752 tablet_service.cc:1467] Processing CreateTablet for tablet d02caa04543d43e086f5cd56ac1ead1a (DEFAULT_TABLE table=default.Table_2275071ba627422c802e5b2506d5594c [id=4abf3b98488543d2933165c2f5a11a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.107052 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d02caa04543d43e086f5cd56ac1ead1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.123247 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_0169be5ee58141f4b36608c1a2d3dc2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.123863 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0169be5ee58141f4b36608c1a2d3dc2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.124814 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9125dd3f9544a54ac373ae0f90cce5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.127655 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_12a6d3a2f70d4920b28aa77c6654b93e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:23.127892 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_89761f48fc8547e8b786c9559485f243"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.128091 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12a6d3a2f70d4920b28aa77c6654b93e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:23.128278 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89761f48fc8547e8b786c9559485f243 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.129277 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_7d31752a7e7246ad9806310039f73c14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:23.129333 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_a78a69e0b1914d25a0f0726d75b66731"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.129714 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a78a69e0b1914d25a0f0726d75b66731 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:23.130434 511 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 2bca7a6cecdc49aeb9aa4ae65beb7757 because the tablet was not found. No further retry: Not found: Tablet not found: 2bca7a6cecdc49aeb9aa4ae65beb7757
W20250112 02:52:23.130672 511 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet aa767148c9a94a0582de4a77052e0867 because the tablet was not found. No further retry: Not found: Tablet not found: aa767148c9a94a0582de4a77052e0867
I20250112 02:52:23.130770 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d02caa04543d43e086f5cd56ac1ead1a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:23.131042 511 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 600249dc30c842ae9ab872f01ff4b6c5 because the tablet was not found. No further retry: Not found: Tablet not found: 600249dc30c842ae9ab872f01ff4b6c5
W20250112 02:52:23.131229 511 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet e56a00ec3f5040b982c555d65f07d34d because the tablet was not found. No further retry: Not found: Tablet not found: e56a00ec3f5040b982c555d65f07d34d
W20250112 02:52:23.131925 511 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 3092eb25d1ac4333a9e5323f67738998 because the tablet was not found. No further retry: Not found: Tablet not found: 3092eb25d1ac4333a9e5323f67738998
W20250112 02:52:23.132124 511 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet d9125dd3f9544a54ac373ae0f90cce5f because the tablet was not found. No further retry: Not found: Tablet not found: d9125dd3f9544a54ac373ae0f90cce5f
W20250112 02:52:23.132438 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d31752a7e7246ad9806310039f73c14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.136387 1178 raft_consensus.cc:491] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:23.136649 1102 heartbeater.cc:502] Master 127.0.104.61:38771 was elected leader, sending a full tablet report...
I20250112 02:52:23.136610 1178 raft_consensus.cc:513] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.138604 1178 leader_election.cc:290] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:23.147648 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cff5dd0596c4809ad4b50b27729b48b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.151324 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ed69c15f3d1448b929c3e9c768deb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.154186 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa767148c9a94a0582de4a77052e0867. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.158141 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3092eb25d1ac4333a9e5323f67738998. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.162523 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cef1a66738c4bb3bad7019ce64f2ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.165048 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c9a803ff19645328107d3b2fabb90c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.171525 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 600249dc30c842ae9ab872f01ff4b6c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.186003 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed69c15f3d1448b929c3e9c768deb1f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:23.189953 991 leader_election.cc:343] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:23.190361 1177 tablet_bootstrap.cc:492] T 600249dc30c842ae9ab872f01ff4b6c5 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:23.194083 1177 tablet_bootstrap.cc:654] T 600249dc30c842ae9ab872f01ff4b6c5 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.195446 1177 log.cc:826] T 600249dc30c842ae9ab872f01ff4b6c5 P 289afc116aa74408aa5bff74e11c50d8: Log is configured to *not* fsync() on all Append() calls
I20250112 02:52:23.195323 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed69c15f3d1448b929c3e9c768deb1f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
W20250112 02:52:23.200217 989 leader_election.cc:343] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:23.200402 989 leader_election.cc:304] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:23.200886 1178 raft_consensus.cc:2743] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:23.207257 1177 tablet_bootstrap.cc:492] T 600249dc30c842ae9ab872f01ff4b6c5 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:23.207553 1177 ts_tablet_manager.cc:1397] T 600249dc30c842ae9ab872f01ff4b6c5 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.011s
I20250112 02:52:23.215960 1177 raft_consensus.cc:357] T 600249dc30c842ae9ab872f01ff4b6c5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.216394 1177 raft_consensus.cc:383] T 600249dc30c842ae9ab872f01ff4b6c5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.216518 1177 raft_consensus.cc:738] T 600249dc30c842ae9ab872f01ff4b6c5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.217434 1177 consensus_queue.cc:260] T 600249dc30c842ae9ab872f01ff4b6c5 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.226303 1177 ts_tablet_manager.cc:1428] T 600249dc30c842ae9ab872f01ff4b6c5 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.019s user 0.008s sys 0.009s
I20250112 02:52:23.230587 1177 tablet_bootstrap.cc:492] T d9125dd3f9544a54ac373ae0f90cce5f P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:23.232101 1177 tablet_bootstrap.cc:654] T d9125dd3f9544a54ac373ae0f90cce5f P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:23.236801 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet e56a00ec3f5040b982c555d65f07d34d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e56a00ec3f5040b982c555d65f07d34d already in progress: creating tablet
I20250112 02:52:23.237491 1177 tablet_bootstrap.cc:492] T d9125dd3f9544a54ac373ae0f90cce5f P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:23.237615 1177 ts_tablet_manager.cc:1397] T d9125dd3f9544a54ac373ae0f90cce5f P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20250112 02:52:23.238426 1177 raft_consensus.cc:357] T d9125dd3f9544a54ac373ae0f90cce5f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.238612 1177 raft_consensus.cc:383] T d9125dd3f9544a54ac373ae0f90cce5f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.238682 1177 raft_consensus.cc:738] T d9125dd3f9544a54ac373ae0f90cce5f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.238907 1177 consensus_queue.cc:260] T d9125dd3f9544a54ac373ae0f90cce5f P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.241374 1177 ts_tablet_manager.cc:1428] T d9125dd3f9544a54ac373ae0f90cce5f P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 02:52:23.244154 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_a7f9e8004af1434d9cef01112b8d1849" } new_table_name: "default.Table_62be2ee0d4744f198098621bf776cf01" modify_external_catalogs: true
I20250112 02:52:23.245494 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_1196bbb623b948c68037129ce66f8241"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.245992 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1196bbb623b948c68037129ce66f8241 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.246210 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_5fe703bbcb87402b87a82e1441569d53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:23.247081 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_77d3bedb6a0c45b1a6b23ed2df9f4593" } new_table_name: "default.Table_1ace3ace3c994601a8102bd18c847fa5" modify_external_catalogs: true
I20250112 02:52:23.247143 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_b29145eb86ed42eabdd10eb4a9026639"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:23.247781 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_2ee966aa96bb4ce5a5cdfb58d2add206"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:23.248750 1177 tablet_bootstrap.cc:492] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:23.250046 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_199725d90df746acb9c1beb33de2ff4b" } new_table_name: "default.Table_1a4e5b8794744193902634c51c0285be" modify_external_catalogs: true
I20250112 02:52:23.250427 1177 tablet_bootstrap.cc:654] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.251199 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_201e44b173ce4633b2ca5ac8a21faf95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.251740 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_201e44b173ce4633b2ca5ac8a21faf95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:23.253628 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ee966aa96bb4ce5a5cdfb58d2add206 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:23.255488 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b29145eb86ed42eabdd10eb4a9026639 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:23.256776 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fe703bbcb87402b87a82e1441569d53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.257798 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu with pid 563
I20250112 02:52:23.258631 1181 tablet_bootstrap.cc:492] T d9125dd3f9544a54ac373ae0f90cce5f P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:23.262513 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_2275071ba627422c802e5b2506d5594c" } new_table_name: "default.Table_9d6d135049de4ecab068bd8ab4d9fad4" modify_external_catalogs: true
I20250112 02:52:23.262622 1181 tablet_bootstrap.cc:654] T d9125dd3f9544a54ac373ae0f90cce5f P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:23.264246 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 2bca7a6cecdc49aeb9aa4ae65beb7757 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2bca7a6cecdc49aeb9aa4ae65beb7757 already in progress: creating tablet
I20250112 02:52:23.264585 1171 tablet_replica.cc:331] T 600249dc30c842ae9ab872f01ff4b6c5 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:23.265798 1171 raft_consensus.cc:2238] T 600249dc30c842ae9ab872f01ff4b6c5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.266234 1171 raft_consensus.cc:2267] T 600249dc30c842ae9ab872f01ff4b6c5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.269412 1181 log.cc:826] T d9125dd3f9544a54ac373ae0f90cce5f P fe36951a83fb4e57b26e187edbcbfe3d: Log is configured to *not* fsync() on all Append() calls
I20250112 02:52:23.270449 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_0169be5ee58141f4b36608c1a2d3dc2f" } new_table_name: "default.Table_0b78c0ecb1f245838a0117009873c0e9" modify_external_catalogs: true
I20250112 02:52:23.270852 1171 ts_tablet_manager.cc:1905] T 600249dc30c842ae9ab872f01ff4b6c5 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.278180 751 tablet_service.cc:1467] Processing CreateTablet for tablet 11afb91c4fde4f6c8204c1cae22cffc5 (DEFAULT_TABLE table=default.Table_a78a69e0b1914d25a0f0726d75b66731 [id=e47433a989db4945973f25468fd1581b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.278227 524 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 9c9a803ff19645328107d3b2fabb90c6 (table default.Table_a7f9e8004af1434d9cef01112b8d1849 [id=bf2df21352c64c069c2a06c878911016]). Expected version 1 got 0
I20250112 02:52:23.278496 524 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet d02caa04543d43e086f5cd56ac1ead1a (table default.Table_2275071ba627422c802e5b2506d5594c [id=4abf3b98488543d2933165c2f5a11a94]). Expected version 1 got 0
I20250112 02:52:23.278712 524 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3cff5dd0596c4809ad4b50b27729b48b (table default.Table_38c977627680467690f8534465803113 [id=05e179e7b80a40eaa0e3d425e84dd7a3]). Expected version 1 got 0
I20250112 02:52:23.278854 524 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 1ed69c15f3d1448b929c3e9c768deb1f (table default.Table_199725d90df746acb9c1beb33de2ff4b [id=f20d22fe09e94ecb8dbe6d4f0a4a8b28]). Expected version 1 got 0
I20250112 02:52:23.278998 524 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 8cef1a66738c4bb3bad7019ce64f2ec2 (table default.Table_77d3bedb6a0c45b1a6b23ed2df9f4593 [id=60486b9e81254d74b8e08a1cadde13bd]). Expected version 1 got 0
I20250112 02:52:23.278982 890 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5b418c17084420b867498fda79fb0a (DEFAULT_TABLE table=default.Table_0169be5ee58141f4b36608c1a2d3dc2f [id=ad0be37e6b6f4b9597dcb4cbd50d5b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.279374 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f5b418c17084420b867498fda79fb0a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:23.279757 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c9a803ff19645328107d3b2fabb90c6 consensus state has no leader
I20250112 02:52:23.280138 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_38c977627680467690f8534465803113" } modify_external_catalogs: true
I20250112 02:52:23.280368 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_38c977627680467690f8534465803113" } modify_external_catalogs: true
I20250112 02:52:23.282016 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5b418c17084420b867498fda79fb0a (DEFAULT_TABLE table=default.Table_0169be5ee58141f4b36608c1a2d3dc2f [id=ad0be37e6b6f4b9597dcb4cbd50d5b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.282363 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f5b418c17084420b867498fda79fb0a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:23.282423 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8cef1a66738c4bb3bad7019ce64f2ec2 consensus state has no leader
I20250112 02:52:23.282773 526 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_12a6d3a2f70d4920b28aa77c6654b93e" } modify_external_catalogs: true
I20250112 02:52:23.282959 526 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_12a6d3a2f70d4920b28aa77c6654b93e" } modify_external_catalogs: true
I20250112 02:52:23.283514 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11afb91c4fde4f6c8204c1cae22cffc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.286672 755 tablet_service.cc:1467] Processing CreateTablet for tablet bb0030637279449fa9c00cdc075fbca9 (DEFAULT_TABLE table=default.Table_12a6d3a2f70d4920b28aa77c6654b93e [id=c77dc91fbb544fb49d244b044ab51c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.286995 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb0030637279449fa9c00cdc075fbca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.291515 760 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5b418c17084420b867498fda79fb0a (DEFAULT_TABLE table=default.Table_0169be5ee58141f4b36608c1a2d3dc2f [id=ad0be37e6b6f4b9597dcb4cbd50d5b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.291891 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f5b418c17084420b867498fda79fb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.292052 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 11afb91c4fde4f6c8204c1cae22cffc5 (DEFAULT_TABLE table=default.Table_a78a69e0b1914d25a0f0726d75b66731 [id=e47433a989db4945973f25468fd1581b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.292452 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11afb91c4fde4f6c8204c1cae22cffc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.295224 1035 tablet_service.cc:1467] Processing CreateTablet for tablet bb0030637279449fa9c00cdc075fbca9 (DEFAULT_TABLE table=default.Table_12a6d3a2f70d4920b28aa77c6654b93e [id=c77dc91fbb544fb49d244b044ab51c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.295601 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb0030637279449fa9c00cdc075fbca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.299208 899 tablet_service.cc:1467] Processing CreateTablet for tablet bb0030637279449fa9c00cdc075fbca9 (DEFAULT_TABLE table=default.Table_12a6d3a2f70d4920b28aa77c6654b93e [id=c77dc91fbb544fb49d244b044ab51c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.299674 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb0030637279449fa9c00cdc075fbca9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:23.302505 1100 heartbeater.cc:643] Failed to heartbeat to 127.0.104.60:39987 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.104.60:39987 (error 108)
W20250112 02:52:23.304209 825 heartbeater.cc:643] Failed to heartbeat to 127.0.104.60:39987 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.104.60:39987 (error 108)
I20250112 02:52:23.306552 894 tablet_service.cc:1467] Processing CreateTablet for tablet 11afb91c4fde4f6c8204c1cae22cffc5 (DEFAULT_TABLE table=default.Table_a78a69e0b1914d25a0f0726d75b66731 [id=e47433a989db4945973f25468fd1581b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.306937 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11afb91c4fde4f6c8204c1cae22cffc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.315805 1173 ts_tablet_manager.cc:1428] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.282s user 0.046s sys 0.073s
I20250112 02:52:23.318127 1177 tablet_bootstrap.cc:492] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:23.318302 1177 ts_tablet_manager.cc:1397] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.070s user 0.021s sys 0.042s
I20250112 02:52:23.318764 1173 tablet_bootstrap.cc:492] T 3092eb25d1ac4333a9e5323f67738998 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:23.320351 1173 tablet_bootstrap.cc:654] T 3092eb25d1ac4333a9e5323f67738998 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:23.320935 1104 negotiation.cc:337] Failed RPC negotiation. Trace:
0112 02:52:23.316404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:39987 (local address 127.0.104.3:39153)
0112 02:52:23.316613 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0112 02:52:23.316631 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0112 02:52:23.316726 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0112 02:52:23.319894 (+ 3168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":109}
I20250112 02:52:23.327186 1181 tablet_bootstrap.cc:492] T d9125dd3f9544a54ac373ae0f90cce5f P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:23.327558 1181 ts_tablet_manager.cc:1397] T d9125dd3f9544a54ac373ae0f90cce5f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.090s user 0.005s sys 0.003s
I20250112 02:52:23.329479 1171 ts_tablet_manager.cc:1918] T 600249dc30c842ae9ab872f01ff4b6c5 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.329666 1171 log.cc:1198] T 600249dc30c842ae9ab872f01ff4b6c5 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/600249dc30c842ae9ab872f01ff4b6c5
I20250112 02:52:23.330067 1171 ts_tablet_manager.cc:1939] T 600249dc30c842ae9ab872f01ff4b6c5 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:23.330518 1177 raft_consensus.cc:357] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.330783 1177 raft_consensus.cc:383] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.330868 1177 raft_consensus.cc:738] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.331627 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 600249dc30c842ae9ab872f01ff4b6c5 (table default.Table_2275071ba627422c802e5b2506d5594c [id=4abf3b98488543d2933165c2f5a11a94]) successfully deleted
I20250112 02:52:23.331789 1171 tablet_replica.cc:331] T d9125dd3f9544a54ac373ae0f90cce5f P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:23.331965 1171 raft_consensus.cc:2238] T d9125dd3f9544a54ac373ae0f90cce5f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.332108 1171 raft_consensus.cc:2267] T d9125dd3f9544a54ac373ae0f90cce5f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:23.332219 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet aa767148c9a94a0582de4a77052e0867 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa767148c9a94a0582de4a77052e0867 already in progress: creating tablet
W20250112 02:52:23.332486 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 3092eb25d1ac4333a9e5323f67738998 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3092eb25d1ac4333a9e5323f67738998 already in progress: creating tablet
I20250112 02:52:23.331194 1177 consensus_queue.cc:260] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.333206 1181 raft_consensus.cc:357] T d9125dd3f9544a54ac373ae0f90cce5f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.334378 1177 ts_tablet_manager.cc:1428] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.016s user 0.001s sys 0.002s
I20250112 02:52:23.334561 1181 raft_consensus.cc:383] T d9125dd3f9544a54ac373ae0f90cce5f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.334683 1177 tablet_bootstrap.cc:492] T e56a00ec3f5040b982c555d65f07d34d P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:23.334790 1181 raft_consensus.cc:738] T d9125dd3f9544a54ac373ae0f90cce5f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.336221 1177 tablet_bootstrap.cc:654] T e56a00ec3f5040b982c555d65f07d34d P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.336146 1181 consensus_queue.cc:260] T d9125dd3f9544a54ac373ae0f90cce5f P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:23.337165 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ed69c15f3d1448b929c3e9c768deb1f consensus state has no leader
I20250112 02:52:23.337874 520 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet d02caa04543d43e086f5cd56ac1ead1a (table default.Table_2275071ba627422c802e5b2506d5594c [id=4abf3b98488543d2933165c2f5a11a94]). Expected version 1 got 0
I20250112 02:52:23.338479 517 master_service.cc:946] ReplaceTablet: received request to replace tablet d02caa04543d43e086f5cd56ac1ead1a from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:23.339834 1171 ts_tablet_manager.cc:1905] T d9125dd3f9544a54ac373ae0f90cce5f P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.340108 1173 tablet_bootstrap.cc:492] T 3092eb25d1ac4333a9e5323f67738998 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:23.340245 1173 ts_tablet_manager.cc:1397] T 3092eb25d1ac4333a9e5323f67738998 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.002s
I20250112 02:52:23.341363 1173 raft_consensus.cc:357] T 3092eb25d1ac4333a9e5323f67738998 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.341552 1173 raft_consensus.cc:383] T 3092eb25d1ac4333a9e5323f67738998 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.341648 1173 raft_consensus.cc:738] T 3092eb25d1ac4333a9e5323f67738998 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
W20250112 02:52:23.341847 512 consensus_peers.cc:487] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 -> Peer dd25f03f25df49178a5c53c636ecf304 (127.0.104.60:39987): Couldn't send request to peer dd25f03f25df49178a5c53c636ecf304. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:23.344406 1173 consensus_queue.cc:260] T 3092eb25d1ac4333a9e5323f67738998 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.344936 1173 ts_tablet_manager.cc:1428] T 3092eb25d1ac4333a9e5323f67738998 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 02:52:23.346170 1171 ts_tablet_manager.cc:1918] T d9125dd3f9544a54ac373ae0f90cce5f P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.346290 1171 log.cc:1198] T d9125dd3f9544a54ac373ae0f90cce5f P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/d9125dd3f9544a54ac373ae0f90cce5f
I20250112 02:52:23.346657 1171 ts_tablet_manager.cc:1939] T d9125dd3f9544a54ac373ae0f90cce5f P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
W20250112 02:52:23.349133 1187 negotiation.cc:337] Failed RPC negotiation. Trace:
0112 02:52:23.304912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:39987 (local address 127.0.104.1:46715)
0112 02:52:23.347893 (+ 42981us) negotiation.cc:107] Waiting for socket to connect
0112 02:52:23.348050 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39987: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":42872,"threads_started":1}
I20250112 02:52:23.350500 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet d9125dd3f9544a54ac373ae0f90cce5f (table default.Table_2275071ba627422c802e5b2506d5594c [id=4abf3b98488543d2933165c2f5a11a94]) successfully deleted
I20250112 02:52:23.350732 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_a78a69e0b1914d25a0f0726d75b66731" } modify_external_catalogs: true
I20250112 02:52:23.351176 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_a78a69e0b1914d25a0f0726d75b66731" } modify_external_catalogs: true
I20250112 02:52:23.351233 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 3cff5dd0596c4809ad4b50b27729b48b
W20250112 02:52:23.350788 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3f5b418c17084420b867498fda79fb0a consensus state has no leader
I20250112 02:52:23.351845 522 master_service.cc:946] ReplaceTablet: received request to replace tablet aa767148c9a94a0582de4a77052e0867 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:23.351912 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_89761f48fc8547e8b786c9559485f243" } modify_external_catalogs: true
I20250112 02:52:23.352037 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_89761f48fc8547e8b786c9559485f243" } modify_external_catalogs: true
I20250112 02:52:23.352825 526 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet bb0030637279449fa9c00cdc075fbca9
I20250112 02:52:23.353411 526 master_service.cc:946] ReplaceTablet: received request to replace tablet 2bca7a6cecdc49aeb9aa4ae65beb7757 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:23.353838 526 master_service.cc:946] ReplaceTablet: received request to replace tablet 600249dc30c842ae9ab872f01ff4b6c5 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:23.353999 518 master_service.cc:946] ReplaceTablet: received request to replace tablet d9125dd3f9544a54ac373ae0f90cce5f from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:23.354377 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_edaa8b50dfa8463e99e05ab8d1d89bc3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.354846 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edaa8b50dfa8463e99e05ab8d1d89bc3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.355821 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet d02caa04543d43e086f5cd56ac1ead1a
I20250112 02:52:23.355911 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cff5dd0596c4809ad4b50b27729b48b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.356262 517 catalog_manager.cc:6596] ReplaceTablet: tablet d02caa04543d43e086f5cd56ac1ead1a deleted and replaced by tablet 5a8e1b96accd459eaed221ded4a453b6
I20250112 02:52:23.352362 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_7d31752a7e7246ad9806310039f73c14" } modify_external_catalogs: true
I20250112 02:52:23.356802 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_7d31752a7e7246ad9806310039f73c14" } modify_external_catalogs: true
I20250112 02:52:23.357451 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cff5dd0596c4809ad4b50b27729b48b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.357740 894 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0030637279449fa9c00cdc075fbca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.357935 890 tablet_service.cc:1514] Processing DeleteTablet for tablet d02caa04543d43e086f5cd56ac1ead1a with delete_type TABLET_DATA_DELETED (replaced by tablet 5a8e1b96accd459eaed221ded4a453b6) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.359056 966 heartbeater.cc:502] Master 127.0.104.61:38771 was elected leader, sending a full tablet report...
W20250112 02:52:23.359200 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 3cff5dd0596c4809ad4b50b27729b48b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cff5dd0596c4809ad4b50b27729b48b already in progress: creating tablet
W20250112 02:52:23.359498 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet bb0030637279449fa9c00cdc075fbca9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb0030637279449fa9c00cdc075fbca9 already in progress: creating tablet
W20250112 02:52:23.359783 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet d02caa04543d43e086f5cd56ac1ead1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d02caa04543d43e086f5cd56ac1ead1a already in progress: creating tablet
W20250112 02:52:23.361317 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 3cff5dd0596c4809ad4b50b27729b48b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cff5dd0596c4809ad4b50b27729b48b already in progress: creating tablet
W20250112 02:52:23.361651 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5a8e1b96accd459eaed221ded4a453b6 has no consensus state
I20250112 02:52:23.354108 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cff5dd0596c4809ad4b50b27729b48b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.355096 760 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0030637279449fa9c00cdc075fbca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.355465 1183 raft_consensus.cc:491] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 02:52:23.363377 524 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet d02caa04543d43e086f5cd56ac1ead1a (table default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet d02caa04543d43e086f5cd56ac1ead1a consensus state has no leader
I20250112 02:52:23.357226 755 tablet_service.cc:1514] Processing DeleteTablet for tablet d02caa04543d43e086f5cd56ac1ead1a with delete_type TABLET_DATA_DELETED (replaced by tablet 5a8e1b96accd459eaed221ded4a453b6) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.360944 1177 tablet_bootstrap.cc:492] T e56a00ec3f5040b982c555d65f07d34d P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:23.363660 1177 ts_tablet_manager.cc:1397] T e56a00ec3f5040b982c555d65f07d34d P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.005s
I20250112 02:52:23.363591 1183 raft_consensus.cc:513] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.364177 1181 ts_tablet_manager.cc:1428] T d9125dd3f9544a54ac373ae0f90cce5f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.036s user 0.019s sys 0.006s
I20250112 02:52:23.364437 1177 raft_consensus.cc:357] T e56a00ec3f5040b982c555d65f07d34d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.364621 1177 raft_consensus.cc:383] T e56a00ec3f5040b982c555d65f07d34d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.364697 1177 raft_consensus.cc:738] T e56a00ec3f5040b982c555d65f07d34d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.364888 1177 consensus_queue.cc:260] T e56a00ec3f5040b982c555d65f07d34d P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.365502 1183 leader_election.cc:290] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:23.365948 1171 tablet_replica.cc:331] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:23.365993 1181 tablet_bootstrap.cc:492] T 1ed69c15f3d1448b929c3e9c768deb1f P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:23.366104 1171 raft_consensus.cc:2238] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.366259 1171 raft_consensus.cc:2267] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:23.367044 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 3cff5dd0596c4809ad4b50b27729b48b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cff5dd0596c4809ad4b50b27729b48b already in progress: creating tablet
W20250112 02:52:23.367269 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet bb0030637279449fa9c00cdc075fbca9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb0030637279449fa9c00cdc075fbca9 already in progress: creating tablet
I20250112 02:52:23.367501 1171 ts_tablet_manager.cc:1905] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.367955 1181 tablet_bootstrap.cc:654] T 1ed69c15f3d1448b929c3e9c768deb1f P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.372268 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet d9125dd3f9544a54ac373ae0f90cce5f with delete_type TABLET_DATA_DELETED (replaced by tablet d02caa04543d43e086f5cd56ac1ead1a) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.372522 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 600249dc30c842ae9ab872f01ff4b6c5 with delete_type TABLET_DATA_DELETED (replaced by tablet d9125dd3f9544a54ac373ae0f90cce5f) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.372851 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 3092eb25d1ac4333a9e5323f67738998 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c9a803ff19645328107d3b2fabb90c6) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.373044 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0030637279449fa9c00cdc075fbca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.373090 1172 tablet_replica.cc:331] T 3092eb25d1ac4333a9e5323f67738998 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:23.373236 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet d02caa04543d43e086f5cd56ac1ead1a with delete_type TABLET_DATA_DELETED (replaced by tablet 5a8e1b96accd459eaed221ded4a453b6) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.373489 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet aa767148c9a94a0582de4a77052e0867 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:22 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.373548 1172 raft_consensus.cc:2238] T 3092eb25d1ac4333a9e5323f67738998 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.373890 1172 raft_consensus.cc:2267] T 3092eb25d1ac4333a9e5323f67738998 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.380049 890 tablet_service.cc:1467] Processing CreateTablet for tablet 22750265b522471680835e7f98475e6e (DEFAULT_TABLE table=default.Table_89761f48fc8547e8b786c9559485f243 [id=c8799d47318d42d9859e36a37a5a9445]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.380410 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22750265b522471680835e7f98475e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.380836 1171 ts_tablet_manager.cc:1918] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.380992 1171 log.cc:1198] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/d02caa04543d43e086f5cd56ac1ead1a
I20250112 02:52:23.381460 1171 ts_tablet_manager.cc:1939] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:23.382758 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02caa04543d43e086f5cd56ac1ead1a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
W20250112 02:52:23.383291 713 leader_election.cc:343] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:23.383498 1172 ts_tablet_manager.cc:1905] T 3092eb25d1ac4333a9e5323f67738998 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.383670 894 tablet_service.cc:1467] Processing CreateTablet for tablet bdaec169cfb5471391fcd3050bb00431 (DEFAULT_TABLE table=default.Table_7d31752a7e7246ad9806310039f73c14 [id=62176e03ce0b4ced96e5083fa32b1826]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.384073 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdaec169cfb5471391fcd3050bb00431. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.384058 1177 ts_tablet_manager.cc:1428] T e56a00ec3f5040b982c555d65f07d34d P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.020s user 0.010s sys 0.002s
I20250112 02:52:23.386811 1177 tablet_bootstrap.cc:492] T aa767148c9a94a0582de4a77052e0867 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:23.387389 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02caa04543d43e086f5cd56ac1ead1a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:23.387807 899 tablet_service.cc:1467] Processing CreateTablet for tablet 2b34fb96bc5140118725af8346ea827f (DEFAULT_TABLE table=default.Table_1196bbb623b948c68037129ce66f8241 [id=059958c4e1c74d8d8e4c029030d85895]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.388181 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b34fb96bc5140118725af8346ea827f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:23.388187 714 leader_election.cc:343] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:23.388334 714 leader_election.cc:304] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:23.388553 1177 tablet_bootstrap.cc:654] T aa767148c9a94a0582de4a77052e0867 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:23.388753 714 raft_consensus.cc:2681] T d02caa04543d43e086f5cd56ac1ead1a P 289afc116aa74408aa5bff74e11c50d8: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 02:52:23.390738 1181 tablet_bootstrap.cc:492] T 1ed69c15f3d1448b929c3e9c768deb1f P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:23.390863 1181 ts_tablet_manager.cc:1397] T 1ed69c15f3d1448b929c3e9c768deb1f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.003s
I20250112 02:52:23.391397 1172 ts_tablet_manager.cc:1918] T 3092eb25d1ac4333a9e5323f67738998 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.391829 1172 log.cc:1198] T 3092eb25d1ac4333a9e5323f67738998 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/3092eb25d1ac4333a9e5323f67738998
W20250112 02:52:23.391814 828 tablet.cc:2367] T e56a00ec3f5040b982c555d65f07d34d P 289afc116aa74408aa5bff74e11c50d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 02:52:23.391844 1181 raft_consensus.cc:357] T 1ed69c15f3d1448b929c3e9c768deb1f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.392033 1181 raft_consensus.cc:383] T 1ed69c15f3d1448b929c3e9c768deb1f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.392139 1181 raft_consensus.cc:738] T 1ed69c15f3d1448b929c3e9c768deb1f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.392336 1181 consensus_queue.cc:260] T 1ed69c15f3d1448b929c3e9c768deb1f P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.392822 1181 ts_tablet_manager.cc:1428] T 1ed69c15f3d1448b929c3e9c768deb1f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:23.392525 1172 ts_tablet_manager.cc:1939] T 3092eb25d1ac4333a9e5323f67738998 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:23.393087 1181 tablet_bootstrap.cc:492] T 8cef1a66738c4bb3bad7019ce64f2ec2 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:23.394848 1181 tablet_bootstrap.cc:654] T 8cef1a66738c4bb3bad7019ce64f2ec2 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.396890 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet d02caa04543d43e086f5cd56ac1ead1a (table default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94]) successfully deleted
I20250112 02:52:23.397954 757 tablet_service.cc:1514] Processing DeleteTablet for tablet e56a00ec3f5040b982c555d65f07d34d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:22 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.398032 751 tablet_service.cc:1467] Processing CreateTablet for tablet 2b34fb96bc5140118725af8346ea827f (DEFAULT_TABLE table=default.Table_1196bbb623b948c68037129ce66f8241 [id=059958c4e1c74d8d8e4c029030d85895]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.398195 755 tablet_service.cc:1467] Processing CreateTablet for tablet 22750265b522471680835e7f98475e6e (DEFAULT_TABLE table=default.Table_89761f48fc8547e8b786c9559485f243 [id=c8799d47318d42d9859e36a37a5a9445]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.398506 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b34fb96bc5140118725af8346ea827f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.399096 760 tablet_service.cc:1467] Processing CreateTablet for tablet bdaec169cfb5471391fcd3050bb00431 (DEFAULT_TABLE table=default.Table_7d31752a7e7246ad9806310039f73c14 [id=62176e03ce0b4ced96e5083fa32b1826]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.399572 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdaec169cfb5471391fcd3050bb00431. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.404717 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22750265b522471680835e7f98475e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.411725 1171 tablet_replica.cc:331] T e56a00ec3f5040b982c555d65f07d34d P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:23.412392 1171 raft_consensus.cc:2238] T e56a00ec3f5040b982c555d65f07d34d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.413293 1173 tablet_bootstrap.cc:492] T 8cef1a66738c4bb3bad7019ce64f2ec2 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:23.413816 1171 raft_consensus.cc:2267] T e56a00ec3f5040b982c555d65f07d34d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.414947 1171 ts_tablet_manager.cc:1905] T e56a00ec3f5040b982c555d65f07d34d P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.416247 1173 tablet_bootstrap.cc:654] T 8cef1a66738c4bb3bad7019ce64f2ec2 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:23.421101 964 heartbeater.cc:643] Failed to heartbeat to 127.0.104.60:39987 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.60:39987: connect: Connection refused (error 111)
I20250112 02:52:23.422247 1171 ts_tablet_manager.cc:1918] T e56a00ec3f5040b982c555d65f07d34d P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.422384 1171 log.cc:1198] T e56a00ec3f5040b982c555d65f07d34d P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/e56a00ec3f5040b982c555d65f07d34d
I20250112 02:52:23.422698 1171 ts_tablet_manager.cc:1939] T e56a00ec3f5040b982c555d65f07d34d P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:23.426981 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet e56a00ec3f5040b982c555d65f07d34d (table default.Table_1ae04ce4866246fc9e5fe0be72adcf64 [id=aa8325dff47c4d95832e0107cd5a17e9]) successfully deleted
I20250112 02:52:23.432900 1173 tablet_bootstrap.cc:492] T 8cef1a66738c4bb3bad7019ce64f2ec2 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:23.433162 1173 ts_tablet_manager.cc:1397] T 8cef1a66738c4bb3bad7019ce64f2ec2 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
W20250112 02:52:23.433013 512 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:42341, user_credentials={real_user=slave}} blocked reactor thread for 60942.2us
W20250112 02:52:23.435086 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet d9125dd3f9544a54ac373ae0f90cce5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9125dd3f9544a54ac373ae0f90cce5f already in progress: creating tablet
I20250112 02:52:23.435222 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 22750265b522471680835e7f98475e6e (DEFAULT_TABLE table=default.Table_89761f48fc8547e8b786c9559485f243 [id=c8799d47318d42d9859e36a37a5a9445]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:23.435348 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 600249dc30c842ae9ab872f01ff4b6c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 600249dc30c842ae9ab872f01ff4b6c5 already in progress: creating tablet
I20250112 02:52:23.435657 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 3092eb25d1ac4333a9e5323f67738998 (table default.Table_62be2ee0d4744f198098621bf776cf01 [id=bf2df21352c64c069c2a06c878911016]) successfully deleted
I20250112 02:52:23.435732 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22750265b522471680835e7f98475e6e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:23.435914 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet bb0030637279449fa9c00cdc075fbca9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb0030637279449fa9c00cdc075fbca9 already in progress: creating tablet
W20250112 02:52:23.436129 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet d02caa04543d43e086f5cd56ac1ead1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d02caa04543d43e086f5cd56ac1ead1a already in progress: creating tablet
W20250112 02:52:23.436314 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet aa767148c9a94a0582de4a77052e0867 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa767148c9a94a0582de4a77052e0867 already in progress: creating tablet
I20250112 02:52:23.437077 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 2b34fb96bc5140118725af8346ea827f (DEFAULT_TABLE table=default.Table_1196bbb623b948c68037129ce66f8241 [id=059958c4e1c74d8d8e4c029030d85895]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.437434 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b34fb96bc5140118725af8346ea827f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.437565 1181 tablet_bootstrap.cc:492] T 8cef1a66738c4bb3bad7019ce64f2ec2 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:23.437723 1181 ts_tablet_manager.cc:1397] T 8cef1a66738c4bb3bad7019ce64f2ec2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.045s user 0.028s sys 0.016s
I20250112 02:52:23.438558 1181 raft_consensus.cc:357] T 8cef1a66738c4bb3bad7019ce64f2ec2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.438773 1181 raft_consensus.cc:383] T 8cef1a66738c4bb3bad7019ce64f2ec2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.438841 1181 raft_consensus.cc:738] T 8cef1a66738c4bb3bad7019ce64f2ec2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.439110 1181 consensus_queue.cc:260] T 8cef1a66738c4bb3bad7019ce64f2ec2 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.439671 1181 ts_tablet_manager.cc:1428] T 8cef1a66738c4bb3bad7019ce64f2ec2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:23.439733 1031 tablet_service.cc:1467] Processing CreateTablet for tablet bdaec169cfb5471391fcd3050bb00431 (DEFAULT_TABLE table=default.Table_7d31752a7e7246ad9806310039f73c14 [id=62176e03ce0b4ced96e5083fa32b1826]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.439939 1181 tablet_bootstrap.cc:492] T 3cff5dd0596c4809ad4b50b27729b48b P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:23.440011 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdaec169cfb5471391fcd3050bb00431. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.441519 1181 tablet_bootstrap.cc:654] T 3cff5dd0596c4809ad4b50b27729b48b P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.442900 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_ae0485ac89ea47e7a8c19a07c5a63730"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.443406 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae0485ac89ea47e7a8c19a07c5a63730 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.449268 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_15a37175ae434dfc9638765263dfb60f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.449765 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15a37175ae434dfc9638765263dfb60f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.450821 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_7e88ecc0aa5e43ffb84af9b8dc6da938"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.451259 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e88ecc0aa5e43ffb84af9b8dc6da938 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.454048 1181 tablet_bootstrap.cc:492] T 3cff5dd0596c4809ad4b50b27729b48b P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:23.454182 1181 ts_tablet_manager.cc:1397] T 3cff5dd0596c4809ad4b50b27729b48b P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20250112 02:52:23.455030 1181 raft_consensus.cc:357] T 3cff5dd0596c4809ad4b50b27729b48b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.455212 1181 raft_consensus.cc:383] T 3cff5dd0596c4809ad4b50b27729b48b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.455299 1181 raft_consensus.cc:738] T 3cff5dd0596c4809ad4b50b27729b48b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.455557 1181 consensus_queue.cc:260] T 3cff5dd0596c4809ad4b50b27729b48b P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.456131 1181 ts_tablet_manager.cc:1428] T 3cff5dd0596c4809ad4b50b27729b48b P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:23.456400 1181 tablet_bootstrap.cc:492] T aa767148c9a94a0582de4a77052e0867 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:23.456707 760 tablet_service.cc:1467] Processing CreateTablet for tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 (DEFAULT_TABLE table=default.Table_2ee966aa96bb4ce5a5cdfb58d2add206 [id=5c4769bfcefd4fe0a0f3c2e315d50c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.456988 894 tablet_service.cc:1467] Processing CreateTablet for tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 (DEFAULT_TABLE table=default.Table_2ee966aa96bb4ce5a5cdfb58d2add206 [id=5c4769bfcefd4fe0a0f3c2e315d50c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.457181 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 325e43fdf2fc4eacb48a5227e2ff3bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.457360 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 325e43fdf2fc4eacb48a5227e2ff3bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.458137 1181 tablet_bootstrap.cc:654] T aa767148c9a94a0582de4a77052e0867 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.459843 890 tablet_service.cc:1467] Processing CreateTablet for tablet 6834be02864543469056a928fafe28ca (DEFAULT_TABLE table=default.Table_5fe703bbcb87402b87a82e1441569d53 [id=a2449caa011d43bfb54aea196629c9f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.460330 899 tablet_service.cc:1467] Processing CreateTablet for tablet c2595782d66e4e4b890c94221b196daa (DEFAULT_TABLE table=default.Table_b29145eb86ed42eabdd10eb4a9026639 [id=8559c454b17b4a00a60689a6a9b19f7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.460649 755 tablet_service.cc:1467] Processing CreateTablet for tablet 6834be02864543469056a928fafe28ca (DEFAULT_TABLE table=default.Table_5fe703bbcb87402b87a82e1441569d53 [id=a2449caa011d43bfb54aea196629c9f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.460749 889 tablet_service.cc:1467] Processing CreateTablet for tablet 2728a022ef0d4952966d397a32356503 (DEFAULT_TABLE table=default.Table_201e44b173ce4633b2ca5ac8a21faf95 [id=9b36703d61a046799cb709c7618b1fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.461061 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6834be02864543469056a928fafe28ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.461138 893 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8e1b96accd459eaed221ded4a453b6 (DEFAULT_TABLE table=default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.462874 1032 tablet_service.cc:1467] Processing CreateTablet for tablet c2595782d66e4e4b890c94221b196daa (DEFAULT_TABLE table=default.Table_b29145eb86ed42eabdd10eb4a9026639 [id=8559c454b17b4a00a60689a6a9b19f7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.463100 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 (DEFAULT_TABLE table=default.Table_2ee966aa96bb4ce5a5cdfb58d2add206 [id=5c4769bfcefd4fe0a0f3c2e315d50c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.463215 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2595782d66e4e4b890c94221b196daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.463579 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 325e43fdf2fc4eacb48a5227e2ff3bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.463737 757 tablet_service.cc:1467] Processing CreateTablet for tablet 2728a022ef0d4952966d397a32356503 (DEFAULT_TABLE table=default.Table_201e44b173ce4633b2ca5ac8a21faf95 [id=9b36703d61a046799cb709c7618b1fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.463740 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 2728a022ef0d4952966d397a32356503 (DEFAULT_TABLE table=default.Table_201e44b173ce4633b2ca5ac8a21faf95 [id=9b36703d61a046799cb709c7618b1fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.464059 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2728a022ef0d4952966d397a32356503. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.464092 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2728a022ef0d4952966d397a32356503. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.464746 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_3079a8d66c2b4cb886a147cb052c49f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.465149 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3079a8d66c2b4cb886a147cb052c49f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.467736 753 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8e1b96accd459eaed221ded4a453b6 (DEFAULT_TABLE table=default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.468043 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a8e1b96accd459eaed221ded4a453b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.472249 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 6834be02864543469056a928fafe28ca (DEFAULT_TABLE table=default.Table_5fe703bbcb87402b87a82e1441569d53 [id=a2449caa011d43bfb54aea196629c9f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.472538 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6834be02864543469056a928fafe28ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.476919 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_0c250ae11537450b996e184089f15e55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.477317 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c250ae11537450b996e184089f15e55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.477388 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8e1b96accd459eaed221ded4a453b6 (DEFAULT_TABLE table=default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.477785 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a8e1b96accd459eaed221ded4a453b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.479854 751 tablet_service.cc:1467] Processing CreateTablet for tablet c2595782d66e4e4b890c94221b196daa (DEFAULT_TABLE table=default.Table_b29145eb86ed42eabdd10eb4a9026639 [id=8559c454b17b4a00a60689a6a9b19f7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.480247 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2595782d66e4e4b890c94221b196daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.482771 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_2ee966aa96bb4ce5a5cdfb58d2add206" } new_table_name: "default.Table_62ec4259810845a3972b753e6e152773" modify_external_catalogs: true
I20250112 02:52:23.483693 1177 tablet_bootstrap.cc:492] T aa767148c9a94a0582de4a77052e0867 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:23.483842 1177 ts_tablet_manager.cc:1397] T aa767148c9a94a0582de4a77052e0867 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.097s user 0.039s sys 0.032s
I20250112 02:52:23.484308 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6834be02864543469056a928fafe28ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.484638 1177 raft_consensus.cc:357] T aa767148c9a94a0582de4a77052e0867 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.484905 1177 raft_consensus.cc:383] T aa767148c9a94a0582de4a77052e0867 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.484979 1177 raft_consensus.cc:738] T aa767148c9a94a0582de4a77052e0867 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.485258 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_201e44b173ce4633b2ca5ac8a21faf95" } new_table_name: "default.Table_6d358fc697c74529822f0b2b70321a02" modify_external_catalogs: true
I20250112 02:52:23.485229 1177 consensus_queue.cc:260] T aa767148c9a94a0582de4a77052e0867 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.487792 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2595782d66e4e4b890c94221b196daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.488085 1177 ts_tablet_manager.cc:1428] T aa767148c9a94a0582de4a77052e0867 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250112 02:52:23.488819 758 tablet_service.cc:1514] Processing DeleteTablet for tablet aa767148c9a94a0582de4a77052e0867 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:22 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.489127 1171 tablet_replica.cc:331] T aa767148c9a94a0582de4a77052e0867 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:23.489323 1171 raft_consensus.cc:2238] T aa767148c9a94a0582de4a77052e0867 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.489509 1171 raft_consensus.cc:2267] T aa767148c9a94a0582de4a77052e0867 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.490489 1177 tablet_bootstrap.cc:492] T 3cff5dd0596c4809ad4b50b27729b48b P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:23.490556 1171 ts_tablet_manager.cc:1905] T aa767148c9a94a0582de4a77052e0867 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.491699 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2728a022ef0d4952966d397a32356503. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.492211 1177 tablet_bootstrap.cc:654] T 3cff5dd0596c4809ad4b50b27729b48b P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.495687 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a8e1b96accd459eaed221ded4a453b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.500288 1171 ts_tablet_manager.cc:1918] T aa767148c9a94a0582de4a77052e0867 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.500424 1171 log.cc:1198] T aa767148c9a94a0582de4a77052e0867 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/aa767148c9a94a0582de4a77052e0867
I20250112 02:52:23.500834 1171 ts_tablet_manager.cc:1939] T aa767148c9a94a0582de4a77052e0867 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:23.502385 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet aa767148c9a94a0582de4a77052e0867 (table default.Table_65d48d9effbb4fd1ab54eb9015244268 [id=3eb02559a95b4223bf8050df99296f16]) successfully deleted
I20250112 02:52:23.504325 1173 raft_consensus.cc:357] T 8cef1a66738c4bb3bad7019ce64f2ec2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.504588 1173 raft_consensus.cc:383] T 8cef1a66738c4bb3bad7019ce64f2ec2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.504658 1173 raft_consensus.cc:738] T 8cef1a66738c4bb3bad7019ce64f2ec2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.504901 1173 consensus_queue.cc:260] T 8cef1a66738c4bb3bad7019ce64f2ec2 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.505417 1173 ts_tablet_manager.cc:1428] T 8cef1a66738c4bb3bad7019ce64f2ec2 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.072s user 0.001s sys 0.001s
I20250112 02:52:23.505723 1173 tablet_bootstrap.cc:492] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:23.509791 1173 tablet_bootstrap.cc:654] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.512585 520 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 1ed69c15f3d1448b929c3e9c768deb1f (table default.Table_1a4e5b8794744193902634c51c0285be [id=f20d22fe09e94ecb8dbe6d4f0a4a8b28]). Expected version 2 got 0
I20250112 02:52:23.515702 1173 tablet_bootstrap.cc:492] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:23.515868 1173 ts_tablet_manager.cc:1397] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20250112 02:52:23.516161 888 tablet_service.cc:1467] Processing CreateTablet for tablet e8220011788a480eb50ee4b670c45739 (DEFAULT_TABLE table=default.Table_edaa8b50dfa8463e99e05ab8d1d89bc3 [id=a6a18be9a78e4959b7d272c447f6fc25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.516553 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8220011788a480eb50ee4b670c45739. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.516749 1173 raft_consensus.cc:357] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.517045 1173 raft_consensus.cc:383] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.517210 1173 raft_consensus.cc:738] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.517441 1173 consensus_queue.cc:260] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.518030 1173 ts_tablet_manager.cc:1428] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:23.518352 1173 tablet_bootstrap.cc:492] T 600249dc30c842ae9ab872f01ff4b6c5 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:23.520126 1173 tablet_bootstrap.cc:654] T 600249dc30c842ae9ab872f01ff4b6c5 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.522810 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 3092eb25d1ac4333a9e5323f67738998 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c9a803ff19645328107d3b2fabb90c6) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.523790 758 tablet_service.cc:1467] Processing CreateTablet for tablet e8220011788a480eb50ee4b670c45739 (DEFAULT_TABLE table=default.Table_edaa8b50dfa8463e99e05ab8d1d89bc3 [id=a6a18be9a78e4959b7d272c447f6fc25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.524250 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8220011788a480eb50ee4b670c45739. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:23.530263 512 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 3092eb25d1ac4333a9e5323f67738998 because the tablet was not found. No further retry: Not found: Tablet not found: 3092eb25d1ac4333a9e5323f67738998
I20250112 02:52:23.533780 1176 raft_consensus.cc:491] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:23.533939 1176 raft_consensus.cc:513] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.534768 1176 leader_election.cc:290] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:23.535400 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cff5dd0596c4809ad4b50b27729b48b" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:23.536149 991 leader_election.cc:343] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:23.536672 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cff5dd0596c4809ad4b50b27729b48b" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:23.537048 919 raft_consensus.cc:2463] T 3cff5dd0596c4809ad4b50b27729b48b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:23.537633 989 leader_election.cc:304] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:23.538064 1176 raft_consensus.cc:2798] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:23.538185 1176 raft_consensus.cc:491] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:23.538276 1176 raft_consensus.cc:3054] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:23.543650 1176 raft_consensus.cc:513] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.545401 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cff5dd0596c4809ad4b50b27729b48b" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
W20250112 02:52:23.545920 991 leader_election.cc:343] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:23.546161 1177 tablet_bootstrap.cc:492] T 3cff5dd0596c4809ad4b50b27729b48b P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:23.546293 1177 ts_tablet_manager.cc:1397] T 3cff5dd0596c4809ad4b50b27729b48b P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.056s user 0.014s sys 0.019s
I20250112 02:52:23.546538 1181 tablet_bootstrap.cc:492] T aa767148c9a94a0582de4a77052e0867 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:23.546706 1181 ts_tablet_manager.cc:1397] T aa767148c9a94a0582de4a77052e0867 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.090s user 0.039s sys 0.027s
I20250112 02:52:23.547048 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cff5dd0596c4809ad4b50b27729b48b" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:23.547042 1177 raft_consensus.cc:357] T 3cff5dd0596c4809ad4b50b27729b48b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.547205 1177 raft_consensus.cc:383] T 3cff5dd0596c4809ad4b50b27729b48b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.547225 919 raft_consensus.cc:3054] T 3cff5dd0596c4809ad4b50b27729b48b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:23.547286 1177 raft_consensus.cc:738] T 3cff5dd0596c4809ad4b50b27729b48b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.547415 1181 raft_consensus.cc:357] T aa767148c9a94a0582de4a77052e0867 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.547629 1181 raft_consensus.cc:383] T aa767148c9a94a0582de4a77052e0867 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.547698 1181 raft_consensus.cc:738] T aa767148c9a94a0582de4a77052e0867 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.547539 1177 consensus_queue.cc:260] T 3cff5dd0596c4809ad4b50b27729b48b P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.547901 1181 consensus_queue.cc:260] T aa767148c9a94a0582de4a77052e0867 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.548408 1181 ts_tablet_manager.cc:1428] T aa767148c9a94a0582de4a77052e0867 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:23.548728 1181 tablet_bootstrap.cc:492] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:23.550068 1181 tablet_bootstrap.cc:654] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.550226 919 raft_consensus.cc:2463] T 3cff5dd0596c4809ad4b50b27729b48b P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:23.550717 1177 ts_tablet_manager.cc:1428] T 3cff5dd0596c4809ad4b50b27729b48b P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:23.551746 989 leader_election.cc:304] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:23.552060 1178 raft_consensus.cc:2798] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:23.552132 1177 tablet_bootstrap.cc:492] T 9c9a803ff19645328107d3b2fabb90c6 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:23.553673 1177 tablet_bootstrap.cc:654] T 9c9a803ff19645328107d3b2fabb90c6 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.559331 1181 tablet_bootstrap.cc:492] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:23.559489 1181 ts_tablet_manager.cc:1397] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.007s
I20250112 02:52:23.560220 1178 raft_consensus.cc:695] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:23.560159 1181 raft_consensus.cc:357] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.560343 1181 raft_consensus.cc:383] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.560429 1181 raft_consensus.cc:738] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.560606 1181 consensus_queue.cc:260] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.560757 1178 consensus_queue.cc:237] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.561050 1181 ts_tablet_manager.cc:1428] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:23.561347 1181 tablet_bootstrap.cc:492] T e56a00ec3f5040b982c555d65f07d34d P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:23.562500 1181 tablet_bootstrap.cc:654] T e56a00ec3f5040b982c555d65f07d34d P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.564004 1181 tablet_bootstrap.cc:492] T e56a00ec3f5040b982c555d65f07d34d P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:23.564126 1181 ts_tablet_manager.cc:1397] T e56a00ec3f5040b982c555d65f07d34d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250112 02:52:23.564695 1181 raft_consensus.cc:357] T e56a00ec3f5040b982c555d65f07d34d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.564877 1181 raft_consensus.cc:383] T e56a00ec3f5040b982c555d65f07d34d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.564950 1181 raft_consensus.cc:738] T e56a00ec3f5040b982c555d65f07d34d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.565263 1181 consensus_queue.cc:260] T e56a00ec3f5040b982c555d65f07d34d P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:23.565552 1103 tablet.cc:2367] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 02:52:23.565964 1181 ts_tablet_manager.cc:1428] T e56a00ec3f5040b982c555d65f07d34d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:23.566248 1181 tablet_bootstrap.cc:492] T d02caa04543d43e086f5cd56ac1ead1a P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:23.567691 1181 tablet_bootstrap.cc:654] T d02caa04543d43e086f5cd56ac1ead1a P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.568564 1176 leader_election.cc:290] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:23.574234 1127 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f4a49a5756b143fcb767c3f41463f8d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:23.574501 1127 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: This master's current role is: LEADER
I20250112 02:52:23.575450 1181 tablet_bootstrap.cc:492] T d02caa04543d43e086f5cd56ac1ead1a P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:23.575428 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_1196bbb623b948c68037129ce66f8241" } new_table_name: "default.Table_5dc0392916be4ca9b7c2d3cbaa1b2dce" modify_external_catalogs: true
I20250112 02:52:23.575618 1181 ts_tablet_manager.cc:1397] T d02caa04543d43e086f5cd56ac1ead1a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.005s
I20250112 02:52:23.576269 1181 raft_consensus.cc:357] T d02caa04543d43e086f5cd56ac1ead1a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.576469 1181 raft_consensus.cc:383] T d02caa04543d43e086f5cd56ac1ead1a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.576561 1181 raft_consensus.cc:738] T d02caa04543d43e086f5cd56ac1ead1a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.576762 1181 consensus_queue.cc:260] T d02caa04543d43e086f5cd56ac1ead1a P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.577037 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_0c250ae11537450b996e184089f15e55" } new_table_name: "default.Table_2bceb576abd94da4a7aca29c4b4fa89a" modify_external_catalogs: true
I20250112 02:52:23.577236 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_fc5d684c7f2049e5b2976823cfb242ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.577848 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc5d684c7f2049e5b2976823cfb242ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.578009 1181 ts_tablet_manager.cc:1428] T d02caa04543d43e086f5cd56ac1ead1a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 02:52:23.578243 1181 tablet_bootstrap.cc:492] T 9c9a803ff19645328107d3b2fabb90c6 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
W20250112 02:52:23.578411 967 tablet.cc:2367] T 1ed69c15f3d1448b929c3e9c768deb1f P fe36951a83fb4e57b26e187edbcbfe3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 02:52:23.579927 1181 tablet_bootstrap.cc:654] T 9c9a803ff19645328107d3b2fabb90c6 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.579885 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cff5dd0596c4809ad4b50b27729b48b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.580112 1171 tablet_replica.cc:331] T 3cff5dd0596c4809ad4b50b27729b48b P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:23.580271 1171 raft_consensus.cc:2238] T 3cff5dd0596c4809ad4b50b27729b48b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.580418 1171 raft_consensus.cc:2267] T 3cff5dd0596c4809ad4b50b27729b48b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.581714 524 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 8cef1a66738c4bb3bad7019ce64f2ec2 (table default.Table_1ace3ace3c994601a8102bd18c847fa5 [id=60486b9e81254d74b8e08a1cadde13bd]). Expected version 2 got 0
I20250112 02:52:23.583966 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cff5dd0596c4809ad4b50b27729b48b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.584188 1172 tablet_replica.cc:331] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:23.584214 1171 ts_tablet_manager.cc:1905] T 3cff5dd0596c4809ad4b50b27729b48b P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.584509 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_5fe703bbcb87402b87a82e1441569d53" } modify_external_catalogs: true
I20250112 02:52:23.584722 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_5fe703bbcb87402b87a82e1441569d53" } modify_external_catalogs: true
W20250112 02:52:23.597394 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 consensus state has no leader
I20250112 02:52:23.597798 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_126d7ad084814c798207d8f2938eebb8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.598344 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_126d7ad084814c798207d8f2938eebb8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:23.599853 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2728a022ef0d4952966d397a32356503 consensus state has no leader
I20250112 02:52:23.600334 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_0f7c10fd7b6245f7a7652d516708fe08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.600847 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f7c10fd7b6245f7a7652d516708fe08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.602459 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 22750265b522471680835e7f98475e6e
I20250112 02:52:23.603248 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_1a4e5b8794744193902634c51c0285be" } modify_external_catalogs: true
I20250112 02:52:23.603420 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_1a4e5b8794744193902634c51c0285be" } modify_external_catalogs: true
I20250112 02:52:23.543891 1029 tablet_service.cc:1467] Processing CreateTablet for tablet e8220011788a480eb50ee4b670c45739 (DEFAULT_TABLE table=default.Table_edaa8b50dfa8463e99e05ab8d1d89bc3 [id=a6a18be9a78e4959b7d272c447f6fc25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.604987 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8220011788a480eb50ee4b670c45739. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.605438 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 22750265b522471680835e7f98475e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:23.606360 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 22750265b522471680835e7f98475e6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22750265b522471680835e7f98475e6e already in progress: creating tablet
I20250112 02:52:23.608042 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 22750265b522471680835e7f98475e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.613894 1171 ts_tablet_manager.cc:1918] T 3cff5dd0596c4809ad4b50b27729b48b P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.614027 1171 log.cc:1198] T 3cff5dd0596c4809ad4b50b27729b48b P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/3cff5dd0596c4809ad4b50b27729b48b
I20250112 02:52:23.619017 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 22750265b522471680835e7f98475e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.620003 1171 ts_tablet_manager.cc:1939] T 3cff5dd0596c4809ad4b50b27729b48b P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:23.620991 1172 raft_consensus.cc:2238] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:23.621872 1172 pending_rounds.cc:62] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6: Trying to abort 1 pending ops.
I20250112 02:52:23.622079 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 3cff5dd0596c4809ad4b50b27729b48b (table default.Table_38c977627680467690f8534465803113 [id=05e179e7b80a40eaa0e3d425e84dd7a3]) successfully deleted
I20250112 02:52:23.622057 1172 pending_rounds.cc:69] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319807229771776 op_type: NO_OP noop_request { }
I20250112 02:52:23.622237 1172 raft_consensus.cc:2883] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:23.622370 1172 raft_consensus.cc:2267] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:23.623399 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 22750265b522471680835e7f98475e6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22750265b522471680835e7f98475e6e already in progress: creating tablet
I20250112 02:52:23.624138 1172 ts_tablet_manager.cc:1905] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.626119 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet bdaec169cfb5471391fcd3050bb00431
I20250112 02:52:23.627998 888 tablet_service.cc:1514] Processing DeleteTablet for tablet bdaec169cfb5471391fcd3050bb00431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:23.628695 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet bdaec169cfb5471391fcd3050bb00431 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdaec169cfb5471391fcd3050bb00431 already in progress: creating tablet
I20250112 02:52:23.629599 1172 ts_tablet_manager.cc:1918] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:23.629719 1172 log.cc:1198] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/3cff5dd0596c4809ad4b50b27729b48b
I20250112 02:52:23.630053 1172 ts_tablet_manager.cc:1939] T 3cff5dd0596c4809ad4b50b27729b48b P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:23.632109 1177 tablet_bootstrap.cc:492] T 9c9a803ff19645328107d3b2fabb90c6 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:23.632298 1177 ts_tablet_manager.cc:1397] T 9c9a803ff19645328107d3b2fabb90c6 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.080s user 0.010s sys 0.025s
I20250112 02:52:23.632324 1173 tablet_bootstrap.cc:492] T 600249dc30c842ae9ab872f01ff4b6c5 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:23.632400 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 3cff5dd0596c4809ad4b50b27729b48b (table default.Table_38c977627680467690f8534465803113 [id=05e179e7b80a40eaa0e3d425e84dd7a3]) successfully deleted
I20250112 02:52:23.632488 1173 ts_tablet_manager.cc:1397] T 600249dc30c842ae9ab872f01ff4b6c5 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.114s user 0.007s sys 0.042s
W20250112 02:52:23.632607 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 22750265b522471680835e7f98475e6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22750265b522471680835e7f98475e6e already in progress: creating tablet
I20250112 02:52:23.633191 1173 raft_consensus.cc:357] T 600249dc30c842ae9ab872f01ff4b6c5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.633185 1177 raft_consensus.cc:357] T 9c9a803ff19645328107d3b2fabb90c6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.633431 1177 raft_consensus.cc:383] T 9c9a803ff19645328107d3b2fabb90c6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.633431 1173 raft_consensus.cc:383] T 600249dc30c842ae9ab872f01ff4b6c5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.633527 1173 raft_consensus.cc:738] T 600249dc30c842ae9ab872f01ff4b6c5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.633531 1177 raft_consensus.cc:738] T 9c9a803ff19645328107d3b2fabb90c6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.633805 1173 consensus_queue.cc:260] T 600249dc30c842ae9ab872f01ff4b6c5 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.633805 1177 consensus_queue.cc:260] T 9c9a803ff19645328107d3b2fabb90c6 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.635111 1177 ts_tablet_manager.cc:1428] T 9c9a803ff19645328107d3b2fabb90c6 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 02:52:23.635457 1177 tablet_bootstrap.cc:492] T 8cef1a66738c4bb3bad7019ce64f2ec2 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:23.637297 1177 tablet_bootstrap.cc:654] T 8cef1a66738c4bb3bad7019ce64f2ec2 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.641149 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet bdaec169cfb5471391fcd3050bb00431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.641428 758 tablet_service.cc:1514] Processing DeleteTablet for tablet bdaec169cfb5471391fcd3050bb00431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:23.642064 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet bdaec169cfb5471391fcd3050bb00431 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdaec169cfb5471391fcd3050bb00431 already in progress: creating tablet
W20250112 02:52:23.642095 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet bdaec169cfb5471391fcd3050bb00431 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdaec169cfb5471391fcd3050bb00431 already in progress: creating tablet
I20250112 02:52:23.642700 1173 ts_tablet_manager.cc:1428] T 600249dc30c842ae9ab872f01ff4b6c5 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20250112 02:52:23.644276 1173 tablet_bootstrap.cc:492] T aa767148c9a94a0582de4a77052e0867 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:23.645042 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b34fb96bc5140118725af8346ea827f consensus state has no leader
I20250112 02:52:23.646732 520 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 9c9a803ff19645328107d3b2fabb90c6 (table default.Table_62be2ee0d4744f198098621bf776cf01 [id=bf2df21352c64c069c2a06c878911016]). Expected version 2 got 0
I20250112 02:52:23.647271 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_3079a8d66c2b4cb886a147cb052c49f3" } new_table_name: "default.Table_f82a967adb9040f69df9732d0d228d8d" modify_external_catalogs: true
I20250112 02:52:23.647766 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cff5dd0596c4809ad4b50b27729b48b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
W20250112 02:52:23.648312 512 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 3cff5dd0596c4809ad4b50b27729b48b because the tablet was not found. No further retry: Not found: Tablet not found: 3cff5dd0596c4809ad4b50b27729b48b
I20250112 02:52:23.649675 1177 tablet_bootstrap.cc:492] T 8cef1a66738c4bb3bad7019ce64f2ec2 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:23.649573 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_809bdba2b6d04bd694d7976176d6e359"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:23.649858 1177 ts_tablet_manager.cc:1397] T 8cef1a66738c4bb3bad7019ce64f2ec2 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20250112 02:52:23.649906 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 600249dc30c842ae9ab872f01ff4b6c5 with delete_type TABLET_DATA_DELETED (replaced by tablet d9125dd3f9544a54ac373ae0f90cce5f) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.650112 1172 tablet_replica.cc:331] T 600249dc30c842ae9ab872f01ff4b6c5 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
W20250112 02:52:23.650179 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_809bdba2b6d04bd694d7976176d6e359 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.650272 1172 raft_consensus.cc:2238] T 600249dc30c842ae9ab872f01ff4b6c5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.650411 1172 raft_consensus.cc:2267] T 600249dc30c842ae9ab872f01ff4b6c5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.650637 1177 raft_consensus.cc:357] T 8cef1a66738c4bb3bad7019ce64f2ec2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.650831 1177 raft_consensus.cc:383] T 8cef1a66738c4bb3bad7019ce64f2ec2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.650913 1177 raft_consensus.cc:738] T 8cef1a66738c4bb3bad7019ce64f2ec2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.651125 1172 ts_tablet_manager.cc:1905] T 600249dc30c842ae9ab872f01ff4b6c5 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.651150 1177 consensus_queue.cc:260] T 8cef1a66738c4bb3bad7019ce64f2ec2 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.651535 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 6834be02864543469056a928fafe28ca
I20250112 02:52:23.652207 524 master_service.cc:946] ReplaceTablet: received request to replace tablet 600249dc30c842ae9ab872f01ff4b6c5 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:23.652674 524 master_service.cc:946] ReplaceTablet: received request to replace tablet 8cef1a66738c4bb3bad7019ce64f2ec2 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:23.653738 1177 ts_tablet_manager.cc:1428] T 8cef1a66738c4bb3bad7019ce64f2ec2 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 02:52:23.654587 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 6834be02864543469056a928fafe28ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:23.655364 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 28abc02b1b21427eae06560b99bcc078 has no consensus state
I20250112 02:52:23.655612 518 master_service.cc:946] ReplaceTablet: received request to replace tablet e56a00ec3f5040b982c555d65f07d34d from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:23.655987 518 master_service.cc:946] ReplaceTablet: received request to replace tablet aa767148c9a94a0582de4a77052e0867 from {username='slave'} at 127.0.0.1:40934
W20250112 02:52:23.656268 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 6834be02864543469056a928fafe28ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6834be02864543469056a928fafe28ca already in progress: creating tablet
I20250112 02:52:23.656354 518 master_service.cc:946] ReplaceTablet: received request to replace tablet 11afb91c4fde4f6c8204c1cae22cffc5 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:23.656811 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_a0ebd60cbffb4360a1890752da449c1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.657259 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0ebd60cbffb4360a1890752da449c1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.657441 1172 ts_tablet_manager.cc:1918] T 600249dc30c842ae9ab872f01ff4b6c5 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.657558 1172 log.cc:1198] T 600249dc30c842ae9ab872f01ff4b6c5 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/600249dc30c842ae9ab872f01ff4b6c5
I20250112 02:52:23.657881 1172 ts_tablet_manager.cc:1939] T 600249dc30c842ae9ab872f01ff4b6c5 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:23.658499 1177 tablet_bootstrap.cc:492] T 1ed69c15f3d1448b929c3e9c768deb1f P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:23.658718 1173 tablet_bootstrap.cc:654] T aa767148c9a94a0582de4a77052e0867 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.659554 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 600249dc30c842ae9ab872f01ff4b6c5 (table default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94]) successfully deleted
I20250112 02:52:23.659950 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 6834be02864543469056a928fafe28ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.660225 1177 tablet_bootstrap.cc:654] T 1ed69c15f3d1448b929c3e9c768deb1f P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:23.660517 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 6834be02864543469056a928fafe28ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6834be02864543469056a928fafe28ca already in progress: creating tablet
I20250112 02:52:23.661218 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 6834be02864543469056a928fafe28ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:23.661795 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 6834be02864543469056a928fafe28ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6834be02864543469056a928fafe28ca already in progress: creating tablet
I20250112 02:52:23.662814 1177 tablet_bootstrap.cc:492] T 1ed69c15f3d1448b929c3e9c768deb1f P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:23.662984 1177 ts_tablet_manager.cc:1397] T 1ed69c15f3d1448b929c3e9c768deb1f P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250112 02:52:23.663900 1177 raft_consensus.cc:357] T 1ed69c15f3d1448b929c3e9c768deb1f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.664103 1177 raft_consensus.cc:383] T 1ed69c15f3d1448b929c3e9c768deb1f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.664191 1177 raft_consensus.cc:738] T 1ed69c15f3d1448b929c3e9c768deb1f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.664429 1177 consensus_queue.cc:260] T 1ed69c15f3d1448b929c3e9c768deb1f P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.668033 1177 ts_tablet_manager.cc:1428] T 1ed69c15f3d1448b929c3e9c768deb1f P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 02:52:23.671772 1177 tablet_bootstrap.cc:492] T 3092eb25d1ac4333a9e5323f67738998 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:23.673393 1177 tablet_bootstrap.cc:654] T 3092eb25d1ac4333a9e5323f67738998 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.675024 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_9ff5918b8b90482c9b89b81bdf7183e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.675504 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ff5918b8b90482c9b89b81bdf7183e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.675674 1177 tablet_bootstrap.cc:492] T 3092eb25d1ac4333a9e5323f67738998 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:23.675798 1177 ts_tablet_manager.cc:1397] T 3092eb25d1ac4333a9e5323f67738998 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250112 02:52:23.676465 1177 raft_consensus.cc:357] T 3092eb25d1ac4333a9e5323f67738998 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.676756 1177 raft_consensus.cc:383] T 3092eb25d1ac4333a9e5323f67738998 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.676888 1177 raft_consensus.cc:738] T 3092eb25d1ac4333a9e5323f67738998 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.677115 1177 consensus_queue.cc:260] T 3092eb25d1ac4333a9e5323f67738998 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.677686 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_7d593fe41a06484c86d98ed0ec028a01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.678141 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d593fe41a06484c86d98ed0ec028a01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.680022 1177 ts_tablet_manager.cc:1428] T 3092eb25d1ac4333a9e5323f67738998 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:23.680344 1173 tablet_bootstrap.cc:492] T aa767148c9a94a0582de4a77052e0867 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:23.680511 1173 ts_tablet_manager.cc:1397] T aa767148c9a94a0582de4a77052e0867 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.002s
I20250112 02:52:23.681099 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 8cef1a66738c4bb3bad7019ce64f2ec2
I20250112 02:52:23.681476 524 catalog_manager.cc:6596] ReplaceTablet: tablet 8cef1a66738c4bb3bad7019ce64f2ec2 deleted and replaced by tablet 71b5f80ed198491780cbe17e6cd3fa15
I20250112 02:52:23.681280 1173 raft_consensus.cc:357] T aa767148c9a94a0582de4a77052e0867 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.681679 1173 raft_consensus.cc:383] T aa767148c9a94a0582de4a77052e0867 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.681835 1173 raft_consensus.cc:738] T aa767148c9a94a0582de4a77052e0867 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.681811 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_01a0b2e48fcd4a769a3d442390622c1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.682214 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01a0b2e48fcd4a769a3d442390622c1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.682229 1173 consensus_queue.cc:260] T aa767148c9a94a0582de4a77052e0867 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.682554 1177 tablet_bootstrap.cc:492] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:23.683292 521 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 9c9a803ff19645328107d3b2fabb90c6 (table default.Table_62be2ee0d4744f198098621bf776cf01 [id=bf2df21352c64c069c2a06c878911016]). Expected version 2 got 0
I20250112 02:52:23.684345 1177 tablet_bootstrap.cc:654] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.684762 1173 ts_tablet_manager.cc:1428] T aa767148c9a94a0582de4a77052e0867 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250112 02:52:23.684980 517 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 9c9a803ff19645328107d3b2fabb90c6 (table default.Table_62be2ee0d4744f198098621bf776cf01 [id=bf2df21352c64c069c2a06c878911016]). Expected version 2 got 0
I20250112 02:52:23.685176 517 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3f5b418c17084420b867498fda79fb0a (table default.Table_0b78c0ecb1f245838a0117009873c0e9 [id=ad0be37e6b6f4b9597dcb4cbd50d5b12]). Expected version 1 got 0
I20250112 02:52:23.685426 1173 tablet_bootstrap.cc:492] T d9125dd3f9544a54ac373ae0f90cce5f P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:23.686986 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cef1a66738c4bb3bad7019ce64f2ec2 with delete_type TABLET_DATA_DELETED (replaced by tablet 71b5f80ed198491780cbe17e6cd3fa15) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.687304 1175 tablet_replica.cc:331] T 8cef1a66738c4bb3bad7019ce64f2ec2 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:23.687480 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 11afb91c4fde4f6c8204c1cae22cffc5
I20250112 02:52:23.687505 1173 tablet_bootstrap.cc:654] T d9125dd3f9544a54ac373ae0f90cce5f P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.687441 1190 raft_consensus.cc:491] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:23.687618 1190 raft_consensus.cc:513] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.687808 1175 raft_consensus.cc:2238] T 8cef1a66738c4bb3bad7019ce64f2ec2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.688017 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 1ed69c15f3d1448b929c3e9c768deb1f
I20250112 02:52:23.688191 1175 raft_consensus.cc:2267] T 8cef1a66738c4bb3bad7019ce64f2ec2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.689677 1190 leader_election.cc:290] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:23.690140 1175 ts_tablet_manager.cc:1905] T 8cef1a66738c4bb3bad7019ce64f2ec2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.694453 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ed69c15f3d1448b929c3e9c768deb1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.694685 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 11afb91c4fde4f6c8204c1cae22cffc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.694802 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 11afb91c4fde4f6c8204c1cae22cffc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.694907 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ed69c15f3d1448b929c3e9c768deb1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.695907 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cef1a66738c4bb3bad7019ce64f2ec2 with delete_type TABLET_DATA_DELETED (replaced by tablet 71b5f80ed198491780cbe17e6cd3fa15) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.696143 1172 tablet_replica.cc:331] T 8cef1a66738c4bb3bad7019ce64f2ec2 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:23.696216 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 11afb91c4fde4f6c8204c1cae22cffc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.696295 1172 raft_consensus.cc:2238] T 8cef1a66738c4bb3bad7019ce64f2ec2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.696478 1172 raft_consensus.cc:2267] T 8cef1a66738c4bb3bad7019ce64f2ec2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.696760 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet aa767148c9a94a0582de4a77052e0867 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:22 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.697681 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ed69c15f3d1448b929c3e9c768deb1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.698168 1172 ts_tablet_manager.cc:1905] T 8cef1a66738c4bb3bad7019ce64f2ec2 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.698696 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cef1a66738c4bb3bad7019ce64f2ec2 with delete_type TABLET_DATA_DELETED (replaced by tablet 71b5f80ed198491780cbe17e6cd3fa15) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.699728 893 tablet_service.cc:1514] Processing DeleteTablet for tablet e56a00ec3f5040b982c555d65f07d34d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:22 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.699993 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cff5dd0596c4809ad4b50b27729b48b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.700214 890 tablet_service.cc:1514] Processing DeleteTablet for tablet aa767148c9a94a0582de4a77052e0867 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:22 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.700402 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 3092eb25d1ac4333a9e5323f67738998 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c9a803ff19645328107d3b2fabb90c6) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.701535 1171 tablet_replica.cc:331] T 1ed69c15f3d1448b929c3e9c768deb1f P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:23.701720 1171 raft_consensus.cc:2238] T 1ed69c15f3d1448b929c3e9c768deb1f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.701916 1171 raft_consensus.cc:2267] T 1ed69c15f3d1448b929c3e9c768deb1f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.702553 1171 ts_tablet_manager.cc:1905] T 1ed69c15f3d1448b929c3e9c768deb1f P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.703668 889 tablet_service.cc:1514] Processing DeleteTablet for tablet d02caa04543d43e086f5cd56ac1ead1a with delete_type TABLET_DATA_DELETED (replaced by tablet 5a8e1b96accd459eaed221ded4a453b6) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.703915 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bca7a6cecdc49aeb9aa4ae65beb7757 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:22 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.704104 890 tablet_service.cc:1514] Processing DeleteTablet for tablet d9125dd3f9544a54ac373ae0f90cce5f with delete_type TABLET_DATA_DELETED (replaced by tablet d02caa04543d43e086f5cd56ac1ead1a) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.704272 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 600249dc30c842ae9ab872f01ff4b6c5 with delete_type TABLET_DATA_DELETED (replaced by tablet d9125dd3f9544a54ac373ae0f90cce5f) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.704528 888 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0030637279449fa9c00cdc075fbca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:23.706488 854 leader_election.cc:343] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 3092eb25d1ac4333a9e5323f67738998
I20250112 02:52:23.707592 1175 ts_tablet_manager.cc:1918] T 8cef1a66738c4bb3bad7019ce64f2ec2 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.707801 1175 log.cc:1198] T 8cef1a66738c4bb3bad7019ce64f2ec2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/8cef1a66738c4bb3bad7019ce64f2ec2
I20250112 02:52:23.708129 1181 tablet_bootstrap.cc:492] T 9c9a803ff19645328107d3b2fabb90c6 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:23.708300 1181 ts_tablet_manager.cc:1397] T 9c9a803ff19645328107d3b2fabb90c6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.130s user 0.032s sys 0.024s
I20250112 02:52:23.708310 1175 ts_tablet_manager.cc:1939] T 8cef1a66738c4bb3bad7019ce64f2ec2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:23.709051 1181 raft_consensus.cc:357] T 9c9a803ff19645328107d3b2fabb90c6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.709295 1181 raft_consensus.cc:383] T 9c9a803ff19645328107d3b2fabb90c6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.709386 1181 raft_consensus.cc:738] T 9c9a803ff19645328107d3b2fabb90c6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.709657 1181 consensus_queue.cc:260] T 9c9a803ff19645328107d3b2fabb90c6 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.710206 1181 ts_tablet_manager.cc:1428] T 9c9a803ff19645328107d3b2fabb90c6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:23.710295 1172 ts_tablet_manager.cc:1918] T 8cef1a66738c4bb3bad7019ce64f2ec2 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.710455 1172 log.cc:1198] T 8cef1a66738c4bb3bad7019ce64f2ec2 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/8cef1a66738c4bb3bad7019ce64f2ec2
I20250112 02:52:23.710538 1181 tablet_bootstrap.cc:492] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:23.713176 1172 ts_tablet_manager.cc:1939] T 8cef1a66738c4bb3bad7019ce64f2ec2 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:23.713341 1181 tablet_bootstrap.cc:654] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.714134 1172 tablet_replica.cc:331] T aa767148c9a94a0582de4a77052e0867 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
W20250112 02:52:23.714278 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09886cd08c714f24a96e4df02bc0e6c8 has no consensus state
I20250112 02:52:23.714337 1172 raft_consensus.cc:2238] T aa767148c9a94a0582de4a77052e0867 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.714542 1172 raft_consensus.cc:2267] T aa767148c9a94a0582de4a77052e0867 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.714988 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 8cef1a66738c4bb3bad7019ce64f2ec2 (table default.Table_1ace3ace3c994601a8102bd18c847fa5 [id=60486b9e81254d74b8e08a1cadde13bd]) successfully deleted
I20250112 02:52:23.715284 1172 ts_tablet_manager.cc:1905] T aa767148c9a94a0582de4a77052e0867 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:23.715306 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 11afb91c4fde4f6c8204c1cae22cffc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11afb91c4fde4f6c8204c1cae22cffc5 already in progress: creating tablet
I20250112 02:52:23.716984 1177 tablet_bootstrap.cc:492] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:23.717267 1177 ts_tablet_manager.cc:1397] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20250112 02:52:23.717993 1177 raft_consensus.cc:357] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:23.718204 1177 raft_consensus.cc:383] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.718278 1177 raft_consensus.cc:738] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.718335 1181 tablet_bootstrap.cc:492] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:23.718557 1181 ts_tablet_manager.cc:1397] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20250112 02:52:23.718577 1177 consensus_queue.cc:260] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:23.719419 1181 raft_consensus.cc:357] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:23.719708 1181 raft_consensus.cc:383] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.719837 1181 raft_consensus.cc:738] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.720075 1181 consensus_queue.cc:260] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:23.720163 1171 ts_tablet_manager.cc:1918] T 1ed69c15f3d1448b929c3e9c768deb1f P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.720257 1171 log.cc:1198] T 1ed69c15f3d1448b929c3e9c768deb1f P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/1ed69c15f3d1448b929c3e9c768deb1f
I20250112 02:52:23.720624 1171 ts_tablet_manager.cc:1939] T 1ed69c15f3d1448b929c3e9c768deb1f P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:23.720642 1181 ts_tablet_manager.cc:1428] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:23.720973 1181 tablet_bootstrap.cc:492] T 600249dc30c842ae9ab872f01ff4b6c5 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:23.721565 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 1ed69c15f3d1448b929c3e9c768deb1f (table default.Table_1a4e5b8794744193902634c51c0285be [id=f20d22fe09e94ecb8dbe6d4f0a4a8b28]) successfully deleted
I20250112 02:52:23.722280 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 8cef1a66738c4bb3bad7019ce64f2ec2 (table default.Table_1ace3ace3c994601a8102bd18c847fa5 [id=60486b9e81254d74b8e08a1cadde13bd]) successfully deleted
I20250112 02:52:23.722316 1175 tablet_replica.cc:331] T 1ed69c15f3d1448b929c3e9c768deb1f P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:23.722522 1175 raft_consensus.cc:2238] T 1ed69c15f3d1448b929c3e9c768deb1f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
W20250112 02:52:23.722640 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 11afb91c4fde4f6c8204c1cae22cffc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11afb91c4fde4f6c8204c1cae22cffc5 already in progress: creating tablet
I20250112 02:52:23.723438 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_6e85e49c707f4021b9a16ca854bb2634"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:23.723692 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_7e76dc846ef449c8b185e81aa37ddcb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.723970 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e85e49c707f4021b9a16ca854bb2634 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:23.724179 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e76dc846ef449c8b185e81aa37ddcb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.722673 1175 raft_consensus.cc:2267] T 1ed69c15f3d1448b929c3e9c768deb1f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.724545 1181 tablet_bootstrap.cc:654] T 600249dc30c842ae9ab872f01ff4b6c5 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:23.727306 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 11afb91c4fde4f6c8204c1cae22cffc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11afb91c4fde4f6c8204c1cae22cffc5 already in progress: creating tablet
I20250112 02:52:23.727723 1177 ts_tablet_manager.cc:1428] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250112 02:52:23.727958 1171 tablet_replica.cc:331] T 8cef1a66738c4bb3bad7019ce64f2ec2 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:23.728272 1171 raft_consensus.cc:2238] T 8cef1a66738c4bb3bad7019ce64f2ec2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.729954 1172 ts_tablet_manager.cc:1918] T aa767148c9a94a0582de4a77052e0867 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.730191 1171 raft_consensus.cc:2267] T 8cef1a66738c4bb3bad7019ce64f2ec2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.731356 1175 ts_tablet_manager.cc:1905] T 1ed69c15f3d1448b929c3e9c768deb1f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.734542 888 tablet_service.cc:1467] Processing CreateTablet for tablet 235e9dd0964c4ea6af7d05ef58acae0d (DEFAULT_TABLE table=default.Table_ae0485ac89ea47e7a8c19a07c5a63730 [id=e659e0cc87684ca79fc3b8241116742a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.734889 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 235e9dd0964c4ea6af7d05ef58acae0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.730468 1177 tablet_bootstrap.cc:492] T bb0030637279449fa9c00cdc075fbca9 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:23.734385 758 tablet_service.cc:1467] Processing CreateTablet for tablet 235e9dd0964c4ea6af7d05ef58acae0d (DEFAULT_TABLE table=default.Table_ae0485ac89ea47e7a8c19a07c5a63730 [id=e659e0cc87684ca79fc3b8241116742a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.734738 751 tablet_service.cc:1467] Processing CreateTablet for tablet 28abc02b1b21427eae06560b99bcc078 (DEFAULT_TABLE table=default.Table_2bceb576abd94da4a7aca29c4b4fa89a [id=703dedbc15f549298f82cbbb62ee27d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.735450 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28abc02b1b21427eae06560b99bcc078. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.736835 1177 tablet_bootstrap.cc:654] T bb0030637279449fa9c00cdc075fbca9 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.737188 1175 ts_tablet_manager.cc:1918] T 1ed69c15f3d1448b929c3e9c768deb1f P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.737289 1175 log.cc:1198] T 1ed69c15f3d1448b929c3e9c768deb1f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/1ed69c15f3d1448b929c3e9c768deb1f
I20250112 02:52:23.737639 1175 ts_tablet_manager.cc:1939] T 1ed69c15f3d1448b929c3e9c768deb1f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:23.738490 1175 tablet_replica.cc:331] T e56a00ec3f5040b982c555d65f07d34d P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:23.738680 1175 raft_consensus.cc:2238] T e56a00ec3f5040b982c555d65f07d34d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.738646 755 tablet_service.cc:1467] Processing CreateTablet for tablet f05d3326545849b9af4519bc66c21069 (DEFAULT_TABLE table=default.Table_15a37175ae434dfc9638765263dfb60f [id=7acec972e5f54ed7b0ae6f3c8d317b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.738832 1175 raft_consensus.cc:2267] T e56a00ec3f5040b982c555d65f07d34d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.739022 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f05d3326545849b9af4519bc66c21069. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.734947 760 tablet_service.cc:1467] Processing CreateTablet for tablet 09886cd08c714f24a96e4df02bc0e6c8 (DEFAULT_TABLE table=default.Table_f82a967adb9040f69df9732d0d228d8d [id=fce07a88937f4f5d8b14483e056bcb4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.739578 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09886cd08c714f24a96e4df02bc0e6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.740497 889 tablet_service.cc:1467] Processing CreateTablet for tablet aa67bf25dc894d7ea96d247a22c14e03 (DEFAULT_TABLE table=default.Table_7e88ecc0aa5e43ffb84af9b8dc6da938 [id=2f9194b28f5044e2a85f9e18fa041746]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.740829 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa67bf25dc894d7ea96d247a22c14e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.741426 894 tablet_service.cc:1467] Processing CreateTablet for tablet f05d3326545849b9af4519bc66c21069 (DEFAULT_TABLE table=default.Table_15a37175ae434dfc9638765263dfb60f [id=7acec972e5f54ed7b0ae6f3c8d317b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.741703 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f05d3326545849b9af4519bc66c21069. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.742261 890 tablet_service.cc:1467] Processing CreateTablet for tablet 09886cd08c714f24a96e4df02bc0e6c8 (DEFAULT_TABLE table=default.Table_f82a967adb9040f69df9732d0d228d8d [id=fce07a88937f4f5d8b14483e056bcb4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.742535 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09886cd08c714f24a96e4df02bc0e6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.742810 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ed69c15f3d1448b929c3e9c768deb1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.743144 893 tablet_service.cc:1467] Processing CreateTablet for tablet 28abc02b1b21427eae06560b99bcc078 (DEFAULT_TABLE table=default.Table_2bceb576abd94da4a7aca29c4b4fa89a [id=703dedbc15f549298f82cbbb62ee27d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.743430 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28abc02b1b21427eae06560b99bcc078. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.735080 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 3092eb25d1ac4333a9e5323f67738998 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c9a803ff19645328107d3b2fabb90c6) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.743825 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 235e9dd0964c4ea6af7d05ef58acae0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.744113 753 tablet_service.cc:1467] Processing CreateTablet for tablet aa67bf25dc894d7ea96d247a22c14e03 (DEFAULT_TABLE table=default.Table_7e88ecc0aa5e43ffb84af9b8dc6da938 [id=2f9194b28f5044e2a85f9e18fa041746]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.744457 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa67bf25dc894d7ea96d247a22c14e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.749976 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 1ed69c15f3d1448b929c3e9c768deb1f (table default.Table_1a4e5b8794744193902634c51c0285be [id=f20d22fe09e94ecb8dbe6d4f0a4a8b28]) successfully deleted
I20250112 02:52:23.755275 520 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 9c9a803ff19645328107d3b2fabb90c6 (table default.Table_62be2ee0d4744f198098621bf776cf01 [id=bf2df21352c64c069c2a06c878911016]). Expected version 2 got 0
I20250112 02:52:23.759084 1171 ts_tablet_manager.cc:1905] T 8cef1a66738c4bb3bad7019ce64f2ec2 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.759936 1175 ts_tablet_manager.cc:1905] T e56a00ec3f5040b982c555d65f07d34d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.752173 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 28abc02b1b21427eae06560b99bcc078 (DEFAULT_TABLE table=default.Table_2bceb576abd94da4a7aca29c4b4fa89a [id=703dedbc15f549298f82cbbb62ee27d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.765362 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28abc02b1b21427eae06560b99bcc078. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.765599 1175 ts_tablet_manager.cc:1918] T e56a00ec3f5040b982c555d65f07d34d P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.765717 1175 log.cc:1198] T e56a00ec3f5040b982c555d65f07d34d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/e56a00ec3f5040b982c555d65f07d34d
I20250112 02:52:23.766113 1175 ts_tablet_manager.cc:1939] T e56a00ec3f5040b982c555d65f07d34d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:23.766993 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet e56a00ec3f5040b982c555d65f07d34d (table default.Table_1ae04ce4866246fc9e5fe0be72adcf64 [id=aa8325dff47c4d95832e0107cd5a17e9]) successfully deleted
I20250112 02:52:23.767230 1175 tablet_replica.cc:331] T 3cff5dd0596c4809ad4b50b27729b48b P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:23.752444 1031 tablet_service.cc:1467] Processing CreateTablet for tablet f05d3326545849b9af4519bc66c21069 (DEFAULT_TABLE table=default.Table_15a37175ae434dfc9638765263dfb60f [id=7acec972e5f54ed7b0ae6f3c8d317b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.767424 1175 raft_consensus.cc:2238] T 3cff5dd0596c4809ad4b50b27729b48b P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.767582 1175 raft_consensus.cc:2267] T 3cff5dd0596c4809ad4b50b27729b48b P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.767796 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f05d3326545849b9af4519bc66c21069. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.752645 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 09886cd08c714f24a96e4df02bc0e6c8 (DEFAULT_TABLE table=default.Table_f82a967adb9040f69df9732d0d228d8d [id=fce07a88937f4f5d8b14483e056bcb4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.772214 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09886cd08c714f24a96e4df02bc0e6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.774048 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cef1a66738c4bb3bad7019ce64f2ec2 with delete_type TABLET_DATA_DELETED (replaced by tablet 71b5f80ed198491780cbe17e6cd3fa15) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.774544 1175 ts_tablet_manager.cc:1905] T 3cff5dd0596c4809ad4b50b27729b48b P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.752444 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 235e9dd0964c4ea6af7d05ef58acae0d (DEFAULT_TABLE table=default.Table_ae0485ac89ea47e7a8c19a07c5a63730 [id=e659e0cc87684ca79fc3b8241116742a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.775719 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 235e9dd0964c4ea6af7d05ef58acae0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.759662 1035 tablet_service.cc:1467] Processing CreateTablet for tablet aa67bf25dc894d7ea96d247a22c14e03 (DEFAULT_TABLE table=default.Table_7e88ecc0aa5e43ffb84af9b8dc6da938 [id=2f9194b28f5044e2a85f9e18fa041746]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.779789 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa67bf25dc894d7ea96d247a22c14e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.780838 1181 tablet_bootstrap.cc:492] T 600249dc30c842ae9ab872f01ff4b6c5 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:23.780853 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc067e895cb4c69af4dedc03533a276 (DEFAULT_TABLE table=default.Table_6e85e49c707f4021b9a16ca854bb2634 [id=03ca66b32af3425892725946ed90c493]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.780972 1181 ts_tablet_manager.cc:1397] T 600249dc30c842ae9ab872f01ff4b6c5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.001s
I20250112 02:52:23.781145 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bc067e895cb4c69af4dedc03533a276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.781667 893 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc067e895cb4c69af4dedc03533a276 (DEFAULT_TABLE table=default.Table_6e85e49c707f4021b9a16ca854bb2634 [id=03ca66b32af3425892725946ed90c493]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.781641 1181 raft_consensus.cc:357] T 600249dc30c842ae9ab872f01ff4b6c5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.781824 1181 raft_consensus.cc:383] T 600249dc30c842ae9ab872f01ff4b6c5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.781895 1181 raft_consensus.cc:738] T 600249dc30c842ae9ab872f01ff4b6c5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.781949 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bc067e895cb4c69af4dedc03533a276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.782133 1181 consensus_queue.cc:260] T 600249dc30c842ae9ab872f01ff4b6c5 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.783286 1181 ts_tablet_manager.cc:1428] T 600249dc30c842ae9ab872f01ff4b6c5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:23.783566 1181 tablet_bootstrap.cc:492] T bb0030637279449fa9c00cdc075fbca9 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:23.785212 1181 tablet_bootstrap.cc:654] T bb0030637279449fa9c00cdc075fbca9 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.787071 1175 ts_tablet_manager.cc:1918] T 3cff5dd0596c4809ad4b50b27729b48b P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.787209 1175 log.cc:1198] T 3cff5dd0596c4809ad4b50b27729b48b P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/3cff5dd0596c4809ad4b50b27729b48b
I20250112 02:52:23.787585 1175 ts_tablet_manager.cc:1939] T 3cff5dd0596c4809ad4b50b27729b48b P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:23.788362 1175 tablet_replica.cc:331] T aa767148c9a94a0582de4a77052e0867 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:23.788548 1175 raft_consensus.cc:2238] T aa767148c9a94a0582de4a77052e0867 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.788712 1175 raft_consensus.cc:2267] T aa767148c9a94a0582de4a77052e0867 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.791992 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 3cff5dd0596c4809ad4b50b27729b48b (table default.Table_38c977627680467690f8534465803113 [id=05e179e7b80a40eaa0e3d425e84dd7a3]) successfully deleted
I20250112 02:52:23.797106 888 tablet_service.cc:1467] Processing CreateTablet for tablet df6ccd04880442d0990c6cb0d27956bd (DEFAULT_TABLE table=default.Table_7d593fe41a06484c86d98ed0ec028a01 [id=a8c0e9da5acc45c3ab7f67427f59e2b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.797420 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df6ccd04880442d0990c6cb0d27956bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.802920 1033 tablet_service.cc:1467] Processing CreateTablet for tablet df6ccd04880442d0990c6cb0d27956bd (DEFAULT_TABLE table=default.Table_7d593fe41a06484c86d98ed0ec028a01 [id=a8c0e9da5acc45c3ab7f67427f59e2b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.803457 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df6ccd04880442d0990c6cb0d27956bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.810248 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 96e201d798c94e448a78e1c8adf41724 (DEFAULT_TABLE table=default.Table_9ff5918b8b90482c9b89b81bdf7183e6 [id=95cdab3e555643fc990b77b2ee8f058a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.810207 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_7e76dc846ef449c8b185e81aa37ddcb9" } new_table_name: "default.Table_c360a95859e5448f9af5f90629a898e4" modify_external_catalogs: true
I20250112 02:52:23.810798 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96e201d798c94e448a78e1c8adf41724. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.810851 888 tablet_service.cc:1467] Processing CreateTablet for tablet 96e201d798c94e448a78e1c8adf41724 (DEFAULT_TABLE table=default.Table_9ff5918b8b90482c9b89b81bdf7183e6 [id=95cdab3e555643fc990b77b2ee8f058a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.811005 1175 ts_tablet_manager.cc:1905] T aa767148c9a94a0582de4a77052e0867 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.811228 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96e201d798c94e448a78e1c8adf41724. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.811400 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_ae0485ac89ea47e7a8c19a07c5a63730" } modify_external_catalogs: true
I20250112 02:52:23.811587 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_ae0485ac89ea47e7a8c19a07c5a63730" } modify_external_catalogs: true
I20250112 02:52:23.815404 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_62ec4259810845a3972b753e6e152773" } new_table_name: "default.Table_1bef29fabb1d4271b04c6c5c3f7c89d5" modify_external_catalogs: true
I20250112 02:52:23.815763 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 49650a419dfd49d5a1547770d3f2fe54 (DEFAULT_TABLE table=default.Table_809bdba2b6d04bd694d7976176d6e359 [id=de27aeff6b3a49b3956498ea5015ff33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.815984 890 tablet_service.cc:1467] Processing CreateTablet for tablet d3e41e307df64109a41b0c1c44d3bda6 (DEFAULT_TABLE table=default.Table_01a0b2e48fcd4a769a3d442390622c1e [id=4d04607bb9004f3cb1871c43d0eb47ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.816310 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3e41e307df64109a41b0c1c44d3bda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.816476 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49650a419dfd49d5a1547770d3f2fe54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.818181 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 3567820f0d4349799b681fdfff0ebad8 (DEFAULT_TABLE table=default.Table_0f7c10fd7b6245f7a7652d516708fe08 [id=18415100e94043e99f9884b853158ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.818516 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3567820f0d4349799b681fdfff0ebad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.820199 894 tablet_service.cc:1467] Processing CreateTablet for tablet 49650a419dfd49d5a1547770d3f2fe54 (DEFAULT_TABLE table=default.Table_809bdba2b6d04bd694d7976176d6e359 [id=de27aeff6b3a49b3956498ea5015ff33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.820518 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49650a419dfd49d5a1547770d3f2fe54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.820638 1029 tablet_service.cc:1467] Processing CreateTablet for tablet d3e41e307df64109a41b0c1c44d3bda6 (DEFAULT_TABLE table=default.Table_01a0b2e48fcd4a769a3d442390622c1e [id=4d04607bb9004f3cb1871c43d0eb47ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.820972 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3e41e307df64109a41b0c1c44d3bda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.824903 899 tablet_service.cc:1467] Processing CreateTablet for tablet 328e307a4b35445f92cbfab24f2ada48 (DEFAULT_TABLE table=default.Table_fc5d684c7f2049e5b2976823cfb242ae [id=c84a7bdac0ed4c2ea7c0599d5f1ca9c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.825217 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 328e307a4b35445f92cbfab24f2ada48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.825394 1175 ts_tablet_manager.cc:1918] T aa767148c9a94a0582de4a77052e0867 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 02:52:23.825467 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 consensus state has no leader
I20250112 02:52:23.825522 1175 log.cc:1198] T aa767148c9a94a0582de4a77052e0867 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/aa767148c9a94a0582de4a77052e0867
I20250112 02:52:23.825870 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 235e9dd0964c4ea6af7d05ef58acae0d
I20250112 02:52:23.825928 1175 ts_tablet_manager.cc:1939] T aa767148c9a94a0582de4a77052e0867 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:23.826954 1172 log.cc:1198] T aa767148c9a94a0582de4a77052e0867 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/aa767148c9a94a0582de4a77052e0867
I20250112 02:52:23.827574 1172 ts_tablet_manager.cc:1939] T aa767148c9a94a0582de4a77052e0867 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:23.827734 895 tablet_service.cc:1467] Processing CreateTablet for tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 (DEFAULT_TABLE table=default.Table_7e76dc846ef449c8b185e81aa37ddcb9 [id=08b0c7e4e0c54ec4a215a390e83a1cf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.828091 896 tablet_service.cc:1467] Processing CreateTablet for tablet 2e9533ea72f6420bb55a53cdfc9125f7 (DEFAULT_TABLE table=default.Table_a0ebd60cbffb4360a1890752da449c1f [id=586b2a68665d402ea251cb9292e19da0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.828474 898 tablet_service.cc:1467] Processing CreateTablet for tablet 71b5f80ed198491780cbe17e6cd3fa15 (DEFAULT_TABLE table=default.Table_1ace3ace3c994601a8102bd18c847fa5 [id=60486b9e81254d74b8e08a1cadde13bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.828161 1035 tablet_service.cc:1467] Processing CreateTablet for tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 (DEFAULT_TABLE table=default.Table_7e76dc846ef449c8b185e81aa37ddcb9 [id=08b0c7e4e0c54ec4a215a390e83a1cf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.830082 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_f82a967adb9040f69df9732d0d228d8d" } modify_external_catalogs: true
I20250112 02:52:23.830302 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_f82a967adb9040f69df9732d0d228d8d" } modify_external_catalogs: true
I20250112 02:52:23.830348 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bca7a6cecdc49aeb9aa4ae65beb7757 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:22 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.830458 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 235e9dd0964c4ea6af7d05ef58acae0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.830766 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet aa767148c9a94a0582de4a77052e0867 (table default.Table_65d48d9effbb4fd1ab54eb9015244268 [id=3eb02559a95b4223bf8050df99296f16]) successfully deleted
I20250112 02:52:23.831135 889 tablet_service.cc:1467] Processing CreateTablet for tablet 3567820f0d4349799b681fdfff0ebad8 (DEFAULT_TABLE table=default.Table_0f7c10fd7b6245f7a7652d516708fe08 [id=18415100e94043e99f9884b853158ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.828727 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 2e9533ea72f6420bb55a53cdfc9125f7 (DEFAULT_TABLE table=default.Table_a0ebd60cbffb4360a1890752da449c1f [id=586b2a68665d402ea251cb9292e19da0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.831943 897 tablet_service.cc:1467] Processing CreateTablet for tablet 5601a9398c9a466582a98967c0e749e2 (DEFAULT_TABLE table=default.Table_126d7ad084814c798207d8f2938eebb8 [id=ed3672fa460a4aa2ae1d3b643ade43e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.831452 755 tablet_service.cc:1467] Processing CreateTablet for tablet 49650a419dfd49d5a1547770d3f2fe54 (DEFAULT_TABLE table=default.Table_809bdba2b6d04bd694d7976176d6e359 [id=de27aeff6b3a49b3956498ea5015ff33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.831739 757 tablet_service.cc:1467] Processing CreateTablet for tablet 328e307a4b35445f92cbfab24f2ada48 (DEFAULT_TABLE table=default.Table_fc5d684c7f2049e5b2976823cfb242ae [id=c84a7bdac0ed4c2ea7c0599d5f1ca9c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.833302 751 tablet_service.cc:1467] Processing CreateTablet for tablet 96e201d798c94e448a78e1c8adf41724 (DEFAULT_TABLE table=default.Table_9ff5918b8b90482c9b89b81bdf7183e6 [id=95cdab3e555643fc990b77b2ee8f058a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.833484 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49650a419dfd49d5a1547770d3f2fe54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.831789 750 tablet_service.cc:1467] Processing CreateTablet for tablet 3567820f0d4349799b681fdfff0ebad8 (DEFAULT_TABLE table=default.Table_0f7c10fd7b6245f7a7652d516708fe08 [id=18415100e94043e99f9884b853158ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.833547 749 tablet_service.cc:1467] Processing CreateTablet for tablet 5601a9398c9a466582a98967c0e749e2 (DEFAULT_TABLE table=default.Table_126d7ad084814c798207d8f2938eebb8 [id=ed3672fa460a4aa2ae1d3b643ade43e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.833703 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96e201d798c94e448a78e1c8adf41724. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.833891 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5601a9398c9a466582a98967c0e749e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.832587 759 tablet_service.cc:1467] Processing CreateTablet for tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 (DEFAULT_TABLE table=default.Table_7e76dc846ef449c8b185e81aa37ddcb9 [id=08b0c7e4e0c54ec4a215a390e83a1cf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.834164 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.834447 758 tablet_service.cc:1467] Processing CreateTablet for tablet df6ccd04880442d0990c6cb0d27956bd (DEFAULT_TABLE table=default.Table_7d593fe41a06484c86d98ed0ec028a01 [id=a8c0e9da5acc45c3ab7f67427f59e2b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.834822 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df6ccd04880442d0990c6cb0d27956bd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:23.836194 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 consensus state has no leader
I20250112 02:52:23.832924 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 235e9dd0964c4ea6af7d05ef58acae0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.832814 756 tablet_service.cc:1467] Processing CreateTablet for tablet 71b5f80ed198491780cbe17e6cd3fa15 (DEFAULT_TABLE table=default.Table_1ace3ace3c994601a8102bd18c847fa5 [id=60486b9e81254d74b8e08a1cadde13bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.833156 754 tablet_service.cc:1467] Processing CreateTablet for tablet 2e9533ea72f6420bb55a53cdfc9125f7 (DEFAULT_TABLE table=default.Table_a0ebd60cbffb4360a1890752da449c1f [id=586b2a68665d402ea251cb9292e19da0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.839910 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e9533ea72f6420bb55a53cdfc9125f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.841024 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.843737 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71b5f80ed198491780cbe17e6cd3fa15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.846874 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e9533ea72f6420bb55a53cdfc9125f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.832067 753 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc067e895cb4c69af4dedc03533a276 (DEFAULT_TABLE table=default.Table_6e85e49c707f4021b9a16ca854bb2634 [id=03ca66b32af3425892725946ed90c493]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.847923 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bc067e895cb4c69af4dedc03533a276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.849828 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 71b5f80ed198491780cbe17e6cd3fa15 (DEFAULT_TABLE table=default.Table_1ace3ace3c994601a8102bd18c847fa5 [id=60486b9e81254d74b8e08a1cadde13bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.850268 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 328e307a4b35445f92cbfab24f2ada48 (DEFAULT_TABLE table=default.Table_fc5d684c7f2049e5b2976823cfb242ae [id=c84a7bdac0ed4c2ea7c0599d5f1ca9c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.851161 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3092eb25d1ac4333a9e5323f67738998" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:23.852133 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71b5f80ed198491780cbe17e6cd3fa15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.854835 1177 tablet_bootstrap.cc:492] T bb0030637279449fa9c00cdc075fbca9 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:23.851450 780 raft_consensus.cc:2463] T 3092eb25d1ac4333a9e5323f67738998 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:23.855320 1177 ts_tablet_manager.cc:1397] T bb0030637279449fa9c00cdc075fbca9 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.125s user 0.030s sys 0.046s
I20250112 02:52:23.855703 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5601a9398c9a466582a98967c0e749e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.855929 1171 ts_tablet_manager.cc:1918] T 8cef1a66738c4bb3bad7019ce64f2ec2 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.856055 1171 log.cc:1198] T 8cef1a66738c4bb3bad7019ce64f2ec2 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/8cef1a66738c4bb3bad7019ce64f2ec2
I20250112 02:52:23.856083 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3567820f0d4349799b681fdfff0ebad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.856171 1177 raft_consensus.cc:357] T bb0030637279449fa9c00cdc075fbca9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.856420 1177 raft_consensus.cc:383] T bb0030637279449fa9c00cdc075fbca9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.856472 1171 ts_tablet_manager.cc:1939] T 8cef1a66738c4bb3bad7019ce64f2ec2 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:23.856552 1177 raft_consensus.cc:738] T bb0030637279449fa9c00cdc075fbca9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.856792 1177 consensus_queue.cc:260] T bb0030637279449fa9c00cdc075fbca9 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.857453 1171 tablet_replica.cc:331] T 3092eb25d1ac4333a9e5323f67738998 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:23.857626 1171 raft_consensus.cc:2238] T 3092eb25d1ac4333a9e5323f67738998 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.857790 1171 raft_consensus.cc:2267] T 3092eb25d1ac4333a9e5323f67738998 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.858495 1171 ts_tablet_manager.cc:1905] T 3092eb25d1ac4333a9e5323f67738998 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.859795 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 328e307a4b35445f92cbfab24f2ada48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.855708 855 leader_election.cc:304] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:23.861320 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_5dc0392916be4ca9b7c2d3cbaa1b2dce" } new_table_name: "default.Table_c2d18d092b5a46f8b59f64aa4a8f02c7" modify_external_catalogs: true
I20250112 02:52:23.861799 1190 raft_consensus.cc:2798] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:23.861969 1190 raft_consensus.cc:491] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:23.862052 1190 raft_consensus.cc:3054] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:23.863826 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3567820f0d4349799b681fdfff0ebad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.864116 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_393b6f264860440ebac3e3148390e317"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:23.834173 760 tablet_service.cc:1467] Processing CreateTablet for tablet d3e41e307df64109a41b0c1c44d3bda6 (DEFAULT_TABLE table=default.Table_01a0b2e48fcd4a769a3d442390622c1e [id=4d04607bb9004f3cb1871c43d0eb47ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.864440 1171 ts_tablet_manager.cc:1918] T 3092eb25d1ac4333a9e5323f67738998 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.864565 1171 log.cc:1198] T 3092eb25d1ac4333a9e5323f67738998 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/3092eb25d1ac4333a9e5323f67738998
I20250112 02:52:23.864596 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3e41e307df64109a41b0c1c44d3bda6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:23.864614 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_393b6f264860440ebac3e3148390e317 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.865005 1171 ts_tablet_manager.cc:1939] T 3092eb25d1ac4333a9e5323f67738998 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:23.865782 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_15a37175ae434dfc9638765263dfb60f" } modify_external_catalogs: true
I20250112 02:52:23.865968 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_15a37175ae434dfc9638765263dfb60f" } modify_external_catalogs: true
I20250112 02:52:23.866209 1171 tablet_replica.cc:331] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:23.866403 1171 raft_consensus.cc:2238] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.866556 1171 raft_consensus.cc:2267] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.867100 1190 raft_consensus.cc:513] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.867228 1171 ts_tablet_manager.cc:1905] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.867161 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_809bdba2b6d04bd694d7976176d6e359" } new_table_name: "default.Table_d561e1740cc545dc8aa4babb76b45c37" modify_external_catalogs: true
I20250112 02:52:23.869248 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 8cef1a66738c4bb3bad7019ce64f2ec2 (table default.Table_1ace3ace3c994601a8102bd18c847fa5 [id=60486b9e81254d74b8e08a1cadde13bd]) successfully deleted
W20250112 02:52:23.869472 513 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 1ed69c15f3d1448b929c3e9c768deb1f because the tablet was not found. No further retry: Not found: Tablet not found: 1ed69c15f3d1448b929c3e9c768deb1f
I20250112 02:52:23.869680 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 3092eb25d1ac4333a9e5323f67738998 (table default.Table_62be2ee0d4744f198098621bf776cf01 [id=bf2df21352c64c069c2a06c878911016]) successfully deleted
I20250112 02:52:23.870726 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 09886cd08c714f24a96e4df02bc0e6c8
I20250112 02:52:23.872943 1171 ts_tablet_manager.cc:1918] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.873060 1171 log.cc:1198] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/2bca7a6cecdc49aeb9aa4ae65beb7757
I20250112 02:52:23.873457 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 09886cd08c714f24a96e4df02bc0e6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.873697 1171 ts_tablet_manager.cc:1939] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:23.874953 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_916fd232386d4e659e4392822be45c8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.875303 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_916fd232386d4e659e4392822be45c8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:23.877388 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b34fb96bc5140118725af8346ea827f consensus state has no leader
I20250112 02:52:23.877282 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_d38ef91002ba4653916c476d1df3a9c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.877753 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d38ef91002ba4653916c476d1df3a9c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.878409 1190 leader_election.cc:290] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:23.879259 855 leader_election.cc:343] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: 3092eb25d1ac4333a9e5323f67738998
I20250112 02:52:23.879877 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 09886cd08c714f24a96e4df02bc0e6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.881115 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 2bca7a6cecdc49aeb9aa4ae65beb7757 (table default.Table_8db6a50cdd8a4fb9a4276fd240476bcd [id=01845880c50e41378a0c744d3b67cdbf]) successfully deleted
W20250112 02:52:23.881325 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 235e9dd0964c4ea6af7d05ef58acae0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 235e9dd0964c4ea6af7d05ef58acae0d already in progress: creating tablet
I20250112 02:52:23.882395 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_091c87de29164fc784158659eeaf545a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.882777 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_091c87de29164fc784158659eeaf545a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:23.882956 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 49650a419dfd49d5a1547770d3f2fe54 consensus state has no leader
I20250112 02:52:23.883096 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet f05d3326545849b9af4519bc66c21069
I20250112 02:52:23.886391 898 tablet_service.cc:1514] Processing DeleteTablet for tablet f05d3326545849b9af4519bc66c21069 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.886552 754 tablet_service.cc:1514] Processing DeleteTablet for tablet f05d3326545849b9af4519bc66c21069 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.887056 1177 ts_tablet_manager.cc:1428] T bb0030637279449fa9c00cdc075fbca9 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.032s user 0.000s sys 0.002s
I20250112 02:52:23.889525 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 09886cd08c714f24a96e4df02bc0e6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.889773 757 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0030637279449fa9c00cdc075fbca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.891353 1177 tablet_bootstrap.cc:492] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:23.892824 1171 tablet_replica.cc:331] T bb0030637279449fa9c00cdc075fbca9 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:23.892963 1171 raft_consensus.cc:2238] T bb0030637279449fa9c00cdc075fbca9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.893124 1171 raft_consensus.cc:2267] T bb0030637279449fa9c00cdc075fbca9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.893667 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_6e85e49c707f4021b9a16ca854bb2634" } modify_external_catalogs: true
I20250112 02:52:23.893811 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_6e85e49c707f4021b9a16ca854bb2634" } modify_external_catalogs: true
W20250112 02:52:23.894138 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 09886cd08c714f24a96e4df02bc0e6c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09886cd08c714f24a96e4df02bc0e6c8 already in progress: creating tablet
W20250112 02:52:23.894423 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f05d3326545849b9af4519bc66c21069 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f05d3326545849b9af4519bc66c21069 already in progress: creating tablet
I20250112 02:52:23.894454 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_2bceb576abd94da4a7aca29c4b4fa89a" } modify_external_catalogs: true
I20250112 02:52:23.894589 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_2bceb576abd94da4a7aca29c4b4fa89a" } modify_external_catalogs: true
I20250112 02:52:23.895900 1177 tablet_bootstrap.cc:654] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.897466 1181 tablet_bootstrap.cc:492] T bb0030637279449fa9c00cdc075fbca9 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:23.897605 1181 ts_tablet_manager.cc:1397] T bb0030637279449fa9c00cdc075fbca9 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.114s user 0.007s sys 0.011s
I20250112 02:52:23.898283 1181 raft_consensus.cc:357] T bb0030637279449fa9c00cdc075fbca9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.898509 1181 raft_consensus.cc:383] T bb0030637279449fa9c00cdc075fbca9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.898627 1181 raft_consensus.cc:738] T bb0030637279449fa9c00cdc075fbca9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.899405 1171 ts_tablet_manager.cc:1905] T bb0030637279449fa9c00cdc075fbca9 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.900272 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 28abc02b1b21427eae06560b99bcc078
I20250112 02:52:23.901005 1181 consensus_queue.cc:260] T bb0030637279449fa9c00cdc075fbca9 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.902382 1181 ts_tablet_manager.cc:1428] T bb0030637279449fa9c00cdc075fbca9 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 02:52:23.902647 1181 tablet_bootstrap.cc:492] T 3f5b418c17084420b867498fda79fb0a P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:23.903152 1175 tablet_replica.cc:331] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:23.903353 1175 raft_consensus.cc:2238] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.903563 1175 raft_consensus.cc:2267] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.904130 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.904793 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet f05d3326545849b9af4519bc66c21069 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.905169 1175 ts_tablet_manager.cc:1905] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:23.905452 854 leader_election.cc:343] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 3092eb25d1ac4333a9e5323f67738998
I20250112 02:52:23.905606 854 leader_election.cc:304] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:23.905817 854 raft_consensus.cc:2681] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 02:52:23.906052 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 4bc067e895cb4c69af4dedc03533a276
I20250112 02:52:23.907159 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bc067e895cb4c69af4dedc03533a276 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:23.907275 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 235e9dd0964c4ea6af7d05ef58acae0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.907385 1181 tablet_bootstrap.cc:654] T 3f5b418c17084420b867498fda79fb0a P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.907790 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bc067e895cb4c69af4dedc03533a276 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.902236 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 5601a9398c9a466582a98967c0e749e2 (DEFAULT_TABLE table=default.Table_126d7ad084814c798207d8f2938eebb8 [id=ed3672fa460a4aa2ae1d3b643ade43e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.908951 1171 ts_tablet_manager.cc:1918] T bb0030637279449fa9c00cdc075fbca9 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.909060 1171 log.cc:1198] T bb0030637279449fa9c00cdc075fbca9 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/bb0030637279449fa9c00cdc075fbca9
I20250112 02:52:23.909174 1173 tablet_bootstrap.cc:492] T d9125dd3f9544a54ac373ae0f90cce5f P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:23.909435 1171 ts_tablet_manager.cc:1939] T bb0030637279449fa9c00cdc075fbca9 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:23.910477 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 28abc02b1b21427eae06560b99bcc078 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.910521 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_e390b5d103344bf89b480cd408388b55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:23.910619 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bc067e895cb4c69af4dedc03533a276 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.910701 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet bb0030637279449fa9c00cdc075fbca9 (table default.Table_12a6d3a2f70d4920b28aa77c6654b93e [id=c77dc91fbb544fb49d244b044ab51c26]) successfully deleted
I20250112 02:52:23.910776 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e9533ea72f6420bb55a53cdfc9125f7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:23.910902 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e390b5d103344bf89b480cd408388b55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:23.911211 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 4bc067e895cb4c69af4dedc03533a276 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4bc067e895cb4c69af4dedc03533a276 already in progress: creating tablet
I20250112 02:52:23.911262 1175 ts_tablet_manager.cc:1918] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.911375 1175 log.cc:1198] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/3092eb25d1ac4333a9e5323f67738998
I20250112 02:52:23.912690 1173 ts_tablet_manager.cc:1397] T d9125dd3f9544a54ac373ae0f90cce5f P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.224s user 0.000s sys 0.013s
I20250112 02:52:23.916072 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_a4b8760d1a934d4292ae52ad8b6879c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.916465 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4b8760d1a934d4292ae52ad8b6879c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.917639 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_7d593fe41a06484c86d98ed0ec028a01" } new_table_name: "default.Table_c667abf579554ea0af764de6515a18c6" modify_external_catalogs: true
I20250112 02:52:23.917883 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet aa767148c9a94a0582de4a77052e0867 (table default.Table_65d48d9effbb4fd1ab54eb9015244268 [id=3eb02559a95b4223bf8050df99296f16]) successfully deleted
I20250112 02:52:23.918685 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_17d23578eb794e99829a550bed4a11d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.919158 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17d23578eb794e99829a550bed4a11d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.922734 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_bd1cdf8c9b3343af8d17a9fd05c77145"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.923099 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd1cdf8c9b3343af8d17a9fd05c77145 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.923512 1175 ts_tablet_manager.cc:1939] T 3092eb25d1ac4333a9e5323f67738998 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:23.924345 1175 tablet_replica.cc:331] T d02caa04543d43e086f5cd56ac1ead1a P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:23.924510 1175 raft_consensus.cc:2238] T d02caa04543d43e086f5cd56ac1ead1a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.924676 1175 raft_consensus.cc:2267] T d02caa04543d43e086f5cd56ac1ead1a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.925292 1175 ts_tablet_manager.cc:1905] T d02caa04543d43e086f5cd56ac1ead1a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.925762 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 3092eb25d1ac4333a9e5323f67738998 (table default.Table_62be2ee0d4744f198098621bf776cf01 [id=bf2df21352c64c069c2a06c878911016]) successfully deleted
I20250112 02:52:23.928004 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 28abc02b1b21427eae06560b99bcc078 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:23.928632 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 28abc02b1b21427eae06560b99bcc078 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28abc02b1b21427eae06560b99bcc078 already in progress: creating tablet
I20250112 02:52:23.928772 1172 tablet_replica.cc:331] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:23.928984 1172 raft_consensus.cc:2238] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.929184 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 328e307a4b35445f92cbfab24f2ada48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.929666 1172 raft_consensus.cc:2267] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.929910 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 28abc02b1b21427eae06560b99bcc078 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:23.930316 1172 ts_tablet_manager.cc:1905] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.930958 1175 ts_tablet_manager.cc:1918] T d02caa04543d43e086f5cd56ac1ead1a P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.931074 1175 log.cc:1198] T d02caa04543d43e086f5cd56ac1ead1a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/d02caa04543d43e086f5cd56ac1ead1a
I20250112 02:52:23.931432 1175 ts_tablet_manager.cc:1939] T d02caa04543d43e086f5cd56ac1ead1a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:23.932339 1175 tablet_replica.cc:331] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:23.932477 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet d02caa04543d43e086f5cd56ac1ead1a (table default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94]) successfully deleted
I20250112 02:52:23.932559 1175 raft_consensus.cc:2238] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.932709 1175 raft_consensus.cc:2267] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.933300 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71b5f80ed198491780cbe17e6cd3fa15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.935802 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5601a9398c9a466582a98967c0e749e2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:23.939138 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df6ccd04880442d0990c6cb0d27956bd consensus state has no leader
I20250112 02:52:23.939939 1175 ts_tablet_manager.cc:1905] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.946126 1175 ts_tablet_manager.cc:1918] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:23.946265 1175 log.cc:1198] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/2bca7a6cecdc49aeb9aa4ae65beb7757
I20250112 02:52:23.946367 753 tablet_service.cc:1467] Processing CreateTablet for tablet fb4c004397ee480d897e8b125d248829 (DEFAULT_TABLE table=default.Table_091c87de29164fc784158659eeaf545a [id=b8c138bea585437c83ee0ed94c008284]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.946626 1175 ts_tablet_manager.cc:1939] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:23.946784 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb4c004397ee480d897e8b125d248829. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.947045 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu
/tmp/dist-test-tasko6SHYo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:39987
--webserver_interface=127.0.104.60
--webserver_port=42831
--builtin_ntp_servers=127.0.104.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42341,127.0.104.61:38771,127.0.104.60:39987
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 02:52:23.947464 889 tablet_service.cc:1467] Processing CreateTablet for tablet fb4c004397ee480d897e8b125d248829 (DEFAULT_TABLE table=default.Table_091c87de29164fc784158659eeaf545a [id=b8c138bea585437c83ee0ed94c008284]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.947553 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 2bca7a6cecdc49aeb9aa4ae65beb7757 (table default.Table_8db6a50cdd8a4fb9a4276fd240476bcd [id=01845880c50e41378a0c744d3b67cdbf]) successfully deleted
I20250112 02:52:23.948058 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb4c004397ee480d897e8b125d248829. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.955119 760 tablet_service.cc:1467] Processing CreateTablet for tablet b6dc79f4d1194f81963a8094ed6b0d2e (DEFAULT_TABLE table=default.Table_393b6f264860440ebac3e3148390e317 [id=824805353d914188ba250e8624065190]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.955502 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6dc79f4d1194f81963a8094ed6b0d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.955564 897 tablet_service.cc:1467] Processing CreateTablet for tablet b6dc79f4d1194f81963a8094ed6b0d2e (DEFAULT_TABLE table=default.Table_393b6f264860440ebac3e3148390e317 [id=824805353d914188ba250e8624065190]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.955920 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6dc79f4d1194f81963a8094ed6b0d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.960160 757 tablet_service.cc:1467] Processing CreateTablet for tablet 9427951452104a7db765437493b21e90 (DEFAULT_TABLE table=default.Table_d38ef91002ba4653916c476d1df3a9c6 [id=fd0dc6c5d5d14251a3e6b52e3d66ed0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.960247 893 tablet_service.cc:1467] Processing CreateTablet for tablet 9427951452104a7db765437493b21e90 (DEFAULT_TABLE table=default.Table_d38ef91002ba4653916c476d1df3a9c6 [id=fd0dc6c5d5d14251a3e6b52e3d66ed0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.960481 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9427951452104a7db765437493b21e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.960572 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9427951452104a7db765437493b21e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.955763 1173 raft_consensus.cc:357] T d9125dd3f9544a54ac373ae0f90cce5f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.960887 1173 raft_consensus.cc:383] T d9125dd3f9544a54ac373ae0f90cce5f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:23.960959 1173 raft_consensus.cc:738] T d9125dd3f9544a54ac373ae0f90cce5f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:23.961172 1173 consensus_queue.cc:260] T d9125dd3f9544a54ac373ae0f90cce5f P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:23.963785 759 tablet_service.cc:1467] Processing CreateTablet for tablet 99715f37fbef47c6860fa1d95e350f20 (DEFAULT_TABLE table=default.Table_916fd232386d4e659e4392822be45c8f [id=d3a0fac8c9234ccc92b4c929c973693d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.964123 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99715f37fbef47c6860fa1d95e350f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.964340 1173 ts_tablet_manager.cc:1428] T d9125dd3f9544a54ac373ae0f90cce5f P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.052s user 0.000s sys 0.002s
I20250112 02:52:23.964622 1173 tablet_bootstrap.cc:492] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:23.964975 889 tablet_service.cc:1467] Processing CreateTablet for tablet 99715f37fbef47c6860fa1d95e350f20 (DEFAULT_TABLE table=default.Table_916fd232386d4e659e4392822be45c8f [id=d3a0fac8c9234ccc92b4c929c973693d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.965262 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99715f37fbef47c6860fa1d95e350f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.966219 1173 tablet_bootstrap.cc:654] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:23.968948 1175 tablet_replica.cc:331] T d9125dd3f9544a54ac373ae0f90cce5f P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:23.969164 1175 raft_consensus.cc:2238] T d9125dd3f9544a54ac373ae0f90cce5f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:23.969389 1175 raft_consensus.cc:2267] T d9125dd3f9544a54ac373ae0f90cce5f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:23.969966 1175 ts_tablet_manager.cc:1905] T d9125dd3f9544a54ac373ae0f90cce5f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:23.970522 1034 tablet_service.cc:1467] Processing CreateTablet for tablet b6dc79f4d1194f81963a8094ed6b0d2e (DEFAULT_TABLE table=default.Table_393b6f264860440ebac3e3148390e317 [id=824805353d914188ba250e8624065190]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.972080 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 9427951452104a7db765437493b21e90 (DEFAULT_TABLE table=default.Table_d38ef91002ba4653916c476d1df3a9c6 [id=fd0dc6c5d5d14251a3e6b52e3d66ed0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.973913 754 tablet_service.cc:1467] Processing CreateTablet for tablet f9186ac46ed441ee9308490e90da0e23 (DEFAULT_TABLE table=default.Table_17d23578eb794e99829a550bed4a11d9 [id=f93b5bb22ec1424ea047383ed9713618]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.974277 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9186ac46ed441ee9308490e90da0e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.974462 897 tablet_service.cc:1467] Processing CreateTablet for tablet f9186ac46ed441ee9308490e90da0e23 (DEFAULT_TABLE table=default.Table_17d23578eb794e99829a550bed4a11d9 [id=f93b5bb22ec1424ea047383ed9713618]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.974778 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9186ac46ed441ee9308490e90da0e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.975844 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 99715f37fbef47c6860fa1d95e350f20 (DEFAULT_TABLE table=default.Table_916fd232386d4e659e4392822be45c8f [id=d3a0fac8c9234ccc92b4c929c973693d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.976538 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet d9125dd3f9544a54ac373ae0f90cce5f with delete_type TABLET_DATA_DELETED (replaced by tablet d02caa04543d43e086f5cd56ac1ead1a) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:23.976999 1031 tablet_service.cc:1467] Processing CreateTablet for tablet fb4c004397ee480d897e8b125d248829 (DEFAULT_TABLE table=default.Table_091c87de29164fc784158659eeaf545a [id=b8c138bea585437c83ee0ed94c008284]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.977442 896 tablet_service.cc:1467] Processing CreateTablet for tablet c69587c5a09d40f0baba5c2f6e8a7b41 (DEFAULT_TABLE table=default.Table_e390b5d103344bf89b480cd408388b55 [id=b5457e4343ee43618c8309e737b69ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.977771 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69587c5a09d40f0baba5c2f6e8a7b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.977808 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6dc79f4d1194f81963a8094ed6b0d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.979952 899 tablet_service.cc:1467] Processing CreateTablet for tablet a47cdc208fbc431b8326364e84432cf5 (DEFAULT_TABLE table=default.Table_bd1cdf8c9b3343af8d17a9fd05c77145 [id=c19f442c5dc7459389b6bac8beda390c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.980181 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a47cdc208fbc431b8326364e84432cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.980391 756 tablet_service.cc:1467] Processing CreateTablet for tablet 97b31ebe035c4552baaffc3133ec73c1 (DEFAULT_TABLE table=default.Table_a4b8760d1a934d4292ae52ad8b6879c5 [id=2cee6e60fe7e454ab6a9f24e2f5c47cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.980577 751 tablet_service.cc:1467] Processing CreateTablet for tablet a47cdc208fbc431b8326364e84432cf5 (DEFAULT_TABLE table=default.Table_bd1cdf8c9b3343af8d17a9fd05c77145 [id=c19f442c5dc7459389b6bac8beda390c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.980727 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97b31ebe035c4552baaffc3133ec73c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.980909 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a47cdc208fbc431b8326364e84432cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.982978 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99715f37fbef47c6860fa1d95e350f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.983770 750 tablet_service.cc:1467] Processing CreateTablet for tablet c69587c5a09d40f0baba5c2f6e8a7b41 (DEFAULT_TABLE table=default.Table_e390b5d103344bf89b480cd408388b55 [id=b5457e4343ee43618c8309e737b69ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.983855 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9427951452104a7db765437493b21e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.984102 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69587c5a09d40f0baba5c2f6e8a7b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.984434 1028 tablet_service.cc:1467] Processing CreateTablet for tablet f9186ac46ed441ee9308490e90da0e23 (DEFAULT_TABLE table=default.Table_17d23578eb794e99829a550bed4a11d9 [id=f93b5bb22ec1424ea047383ed9713618]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.985121 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9186ac46ed441ee9308490e90da0e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.987742 893 tablet_service.cc:1467] Processing CreateTablet for tablet 97b31ebe035c4552baaffc3133ec73c1 (DEFAULT_TABLE table=default.Table_a4b8760d1a934d4292ae52ad8b6879c5 [id=2cee6e60fe7e454ab6a9f24e2f5c47cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:23.987957 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97b31ebe035c4552baaffc3133ec73c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:23.989791 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_65f28b8b506644058434fb4736a80acc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.990327 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65f28b8b506644058434fb4736a80acc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.992036 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_99a3cdac72034a8d90eca921ad5970e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:23.992483 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99a3cdac72034a8d90eca921ad5970e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:23.995232 524 master_service.cc:946] ReplaceTablet: received request to replace tablet e8220011788a480eb50ee4b670c45739 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:23.997747 520 master_service.cc:946] ReplaceTablet: received request to replace tablet 9c9a803ff19645328107d3b2fabb90c6 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:23.999032 525 master_service.cc:946] ReplaceTablet: received request to replace tablet d9125dd3f9544a54ac373ae0f90cce5f from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:24.006508 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb4c004397ee480d897e8b125d248829. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.008126 1175 ts_tablet_manager.cc:1918] T d9125dd3f9544a54ac373ae0f90cce5f P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.008265 1175 log.cc:1198] T d9125dd3f9544a54ac373ae0f90cce5f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/d9125dd3f9544a54ac373ae0f90cce5f
I20250112 02:52:24.008710 1175 ts_tablet_manager.cc:1939] T d9125dd3f9544a54ac373ae0f90cce5f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:24.009769 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 9c9a803ff19645328107d3b2fabb90c6
I20250112 02:52:24.009862 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet e8220011788a480eb50ee4b670c45739
I20250112 02:52:24.010210 524 catalog_manager.cc:6596] ReplaceTablet: tablet e8220011788a480eb50ee4b670c45739 deleted and replaced by tablet b885247d2cb449e0a29f6f5d81518589
I20250112 02:52:24.010767 753 tablet_service.cc:1514] Processing DeleteTablet for tablet e8220011788a480eb50ee4b670c45739 with delete_type TABLET_DATA_DELETED (replaced by tablet b885247d2cb449e0a29f6f5d81518589) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:24.011423 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet e8220011788a480eb50ee4b670c45739 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8220011788a480eb50ee4b670c45739 already in progress: creating tablet
I20250112 02:52:24.011924 520 catalog_manager.cc:6596] ReplaceTablet: tablet 9c9a803ff19645328107d3b2fabb90c6 deleted and replaced by tablet a1a0a13696fc41ccb28218f1275a4b5d
I20250112 02:52:24.013077 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c9a803ff19645328107d3b2fabb90c6 with delete_type TABLET_DATA_DELETED (replaced by tablet a1a0a13696fc41ccb28218f1275a4b5d) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.013332 1171 tablet_replica.cc:331] T 9c9a803ff19645328107d3b2fabb90c6 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
W20250112 02:52:24.013509 512 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 8cef1a66738c4bb3bad7019ce64f2ec2 (table default.Table_1ace3ace3c994601a8102bd18c847fa5 [id=60486b9e81254d74b8e08a1cadde13bd], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8cef1a66738c4bb3bad7019ce64f2ec2 consensus state has no leader
I20250112 02:52:24.013504 1171 raft_consensus.cc:2238] T 9c9a803ff19645328107d3b2fabb90c6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.013664 1171 raft_consensus.cc:2267] T 9c9a803ff19645328107d3b2fabb90c6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.014178 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c9a803ff19645328107d3b2fabb90c6 with delete_type TABLET_DATA_DELETED (replaced by tablet a1a0a13696fc41ccb28218f1275a4b5d) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.014431 1171 ts_tablet_manager.cc:1905] T 9c9a803ff19645328107d3b2fabb90c6 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.014431 893 tablet_service.cc:1514] Processing DeleteTablet for tablet e8220011788a480eb50ee4b670c45739 with delete_type TABLET_DATA_DELETED (replaced by tablet b885247d2cb449e0a29f6f5d81518589) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.012832 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 97b31ebe035c4552baaffc3133ec73c1 (DEFAULT_TABLE table=default.Table_a4b8760d1a934d4292ae52ad8b6879c5 [id=2cee6e60fe7e454ab6a9f24e2f5c47cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.016080 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97b31ebe035c4552baaffc3133ec73c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.003682 1026 tablet_service.cc:1467] Processing CreateTablet for tablet c69587c5a09d40f0baba5c2f6e8a7b41 (DEFAULT_TABLE table=default.Table_e390b5d103344bf89b480cd408388b55 [id=b5457e4343ee43618c8309e737b69ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.020282 1171 ts_tablet_manager.cc:1918] T 9c9a803ff19645328107d3b2fabb90c6 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.020437 1171 log.cc:1198] T 9c9a803ff19645328107d3b2fabb90c6 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/9c9a803ff19645328107d3b2fabb90c6
I20250112 02:52:24.020797 1171 ts_tablet_manager.cc:1939] T 9c9a803ff19645328107d3b2fabb90c6 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:24.021760 750 tablet_service.cc:1467] Processing CreateTablet for tablet 54033a747d5a49b0a1c16f138158f596 (DEFAULT_TABLE table=default.Table_65f28b8b506644058434fb4736a80acc [id=c11cab752ad14d53b9b09fbaaf9b6f36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.022066 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 9c9a803ff19645328107d3b2fabb90c6 (table default.Table_62be2ee0d4744f198098621bf776cf01 [id=bf2df21352c64c069c2a06c878911016]) successfully deleted
I20250112 02:52:24.022179 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54033a747d5a49b0a1c16f138158f596. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.023015 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet d9125dd3f9544a54ac373ae0f90cce5f (table default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94]) successfully deleted
I20250112 02:52:24.023347 1175 tablet_replica.cc:331] T 600249dc30c842ae9ab872f01ff4b6c5 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:24.023540 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69587c5a09d40f0baba5c2f6e8a7b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.023694 1175 raft_consensus.cc:2238] T 600249dc30c842ae9ab872f01ff4b6c5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.023896 1175 raft_consensus.cc:2267] T 600249dc30c842ae9ab872f01ff4b6c5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.024665 1175 ts_tablet_manager.cc:1905] T 600249dc30c842ae9ab872f01ff4b6c5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.028896 523 master_service.cc:946] ReplaceTablet: received request to replace tablet bb0030637279449fa9c00cdc075fbca9 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:24.029129 1173 tablet_bootstrap.cc:492] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:24.029274 1173 ts_tablet_manager.cc:1397] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.065s user 0.000s sys 0.002s
I20250112 02:52:24.029577 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_fc5d684c7f2049e5b2976823cfb242ae" } modify_external_catalogs: true
I20250112 02:52:24.029858 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_fc5d684c7f2049e5b2976823cfb242ae" } modify_external_catalogs: true
I20250112 02:52:24.030043 1173 raft_consensus.cc:357] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.030249 1173 raft_consensus.cc:383] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.030333 1173 raft_consensus.cc:738] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.030611 1173 consensus_queue.cc:260] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.031232 1175 ts_tablet_manager.cc:1918] T 600249dc30c842ae9ab872f01ff4b6c5 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.031401 1175 log.cc:1198] T 600249dc30c842ae9ab872f01ff4b6c5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/600249dc30c842ae9ab872f01ff4b6c5
I20250112 02:52:24.031805 1175 ts_tablet_manager.cc:1939] T 600249dc30c842ae9ab872f01ff4b6c5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:24.032840 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 600249dc30c842ae9ab872f01ff4b6c5 (table default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94]) successfully deleted
I20250112 02:52:24.033176 1175 tablet_replica.cc:331] T bb0030637279449fa9c00cdc075fbca9 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:24.033427 1175 raft_consensus.cc:2238] T bb0030637279449fa9c00cdc075fbca9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.033643 1175 raft_consensus.cc:2267] T bb0030637279449fa9c00cdc075fbca9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.034471 1175 ts_tablet_manager.cc:1905] T bb0030637279449fa9c00cdc075fbca9 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.036187 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_99a3cdac72034a8d90eca921ad5970e7" } new_table_name: "default.Table_c9cd1883fedc45e89202fd92fbbd0e50" modify_external_catalogs: true
I20250112 02:52:24.038036 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_a4b8760d1a934d4292ae52ad8b6879c5" } new_table_name: "default.Table_c8b1e93e51184d488f851b546d2aaaf7" modify_external_catalogs: true
I20250112 02:52:24.039958 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 328e307a4b35445f92cbfab24f2ada48
I20250112 02:52:24.042104 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 328e307a4b35445f92cbfab24f2ada48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.043035 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 328e307a4b35445f92cbfab24f2ada48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:24.043599 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 328e307a4b35445f92cbfab24f2ada48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 328e307a4b35445f92cbfab24f2ada48 already in progress: creating tablet
I20250112 02:52:24.046846 1181 tablet_bootstrap.cc:492] T 3f5b418c17084420b867498fda79fb0a P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:24.047013 1181 ts_tablet_manager.cc:1397] T 3f5b418c17084420b867498fda79fb0a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.144s user 0.028s sys 0.057s
I20250112 02:52:24.047950 520 master_service.cc:946] ReplaceTablet: received request to replace tablet 22750265b522471680835e7f98475e6e from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:24.049495 1181 raft_consensus.cc:357] T 3f5b418c17084420b867498fda79fb0a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.049746 1181 raft_consensus.cc:383] T 3f5b418c17084420b867498fda79fb0a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.049825 1181 raft_consensus.cc:738] T 3f5b418c17084420b867498fda79fb0a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.050086 1181 consensus_queue.cc:260] T 3f5b418c17084420b867498fda79fb0a P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.050184 893 tablet_service.cc:1467] Processing CreateTablet for tablet 54033a747d5a49b0a1c16f138158f596 (DEFAULT_TABLE table=default.Table_65f28b8b506644058434fb4736a80acc [id=c11cab752ad14d53b9b09fbaaf9b6f36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.050554 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54033a747d5a49b0a1c16f138158f596. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.050560 1181 ts_tablet_manager.cc:1428] T 3f5b418c17084420b867498fda79fb0a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:24.051204 1177 tablet_bootstrap.cc:492] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:24.051281 1181 tablet_bootstrap.cc:492] T 11afb91c4fde4f6c8204c1cae22cffc5 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:24.051342 1177 ts_tablet_manager.cc:1397] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.160s user 0.002s sys 0.007s
I20250112 02:52:24.052093 1177 raft_consensus.cc:357] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.052287 1177 raft_consensus.cc:383] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.052356 1177 raft_consensus.cc:738] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.052484 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 328e307a4b35445f92cbfab24f2ada48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.052762 1177 consensus_queue.cc:260] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.053081 1181 tablet_bootstrap.cc:654] T 11afb91c4fde4f6c8204c1cae22cffc5 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.055298 1177 ts_tablet_manager.cc:1428] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:24.056452 1177 tablet_bootstrap.cc:492] T 11afb91c4fde4f6c8204c1cae22cffc5 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:24.056761 525 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3f5b418c17084420b867498fda79fb0a (table default.Table_0b78c0ecb1f245838a0117009873c0e9 [id=ad0be37e6b6f4b9597dcb4cbd50d5b12]). Expected version 1 got 0
I20250112 02:52:24.059489 525 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3f5b418c17084420b867498fda79fb0a (table default.Table_0b78c0ecb1f245838a0117009873c0e9 [id=ad0be37e6b6f4b9597dcb4cbd50d5b12]). Expected version 1 got 0
W20250112 02:52:24.060256 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 97b31ebe035c4552baaffc3133ec73c1 consensus state has no leader
I20250112 02:52:24.061368 517 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3f5b418c17084420b867498fda79fb0a (table default.Table_0b78c0ecb1f245838a0117009873c0e9 [id=ad0be37e6b6f4b9597dcb4cbd50d5b12]). Expected version 1 got 0
W20250112 02:52:24.063635 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f6803d0f553b4204b415b2734286bf4a has no consensus state
I20250112 02:52:24.064201 1175 ts_tablet_manager.cc:1918] T bb0030637279449fa9c00cdc075fbca9 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.066099 1177 tablet_bootstrap.cc:654] T 11afb91c4fde4f6c8204c1cae22cffc5 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.045542 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet e8220011788a480eb50ee4b670c45739 with delete_type TABLET_DATA_DELETED (replaced by tablet b885247d2cb449e0a29f6f5d81518589) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.069020 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 54033a747d5a49b0a1c16f138158f596 (DEFAULT_TABLE table=default.Table_65f28b8b506644058434fb4736a80acc [id=c11cab752ad14d53b9b09fbaaf9b6f36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.069221 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c9a803ff19645328107d3b2fabb90c6 with delete_type TABLET_DATA_DELETED (replaced by tablet a1a0a13696fc41ccb28218f1275a4b5d) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.067737 1181 tablet_bootstrap.cc:492] T 11afb91c4fde4f6c8204c1cae22cffc5 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:24.071563 1175 log.cc:1198] T bb0030637279449fa9c00cdc075fbca9 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/bb0030637279449fa9c00cdc075fbca9
I20250112 02:52:24.071594 1181 ts_tablet_manager.cc:1397] T 11afb91c4fde4f6c8204c1cae22cffc5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.003s
I20250112 02:52:24.072047 1175 ts_tablet_manager.cc:1939] T bb0030637279449fa9c00cdc075fbca9 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:24.072636 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54033a747d5a49b0a1c16f138158f596. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.072528 1181 raft_consensus.cc:357] T 11afb91c4fde4f6c8204c1cae22cffc5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.072768 1181 raft_consensus.cc:383] T 11afb91c4fde4f6c8204c1cae22cffc5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.072865 1181 raft_consensus.cc:738] T 11afb91c4fde4f6c8204c1cae22cffc5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.073089 1181 consensus_queue.cc:260] T 11afb91c4fde4f6c8204c1cae22cffc5 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.073246 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet bb0030637279449fa9c00cdc075fbca9 (table default.Table_12a6d3a2f70d4920b28aa77c6654b93e [id=c77dc91fbb544fb49d244b044ab51c26]) successfully deleted
I20250112 02:52:24.073601 1181 ts_tablet_manager.cc:1428] T 11afb91c4fde4f6c8204c1cae22cffc5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:24.073807 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_7e88ecc0aa5e43ffb84af9b8dc6da938" } modify_external_catalogs: true
I20250112 02:52:24.073943 1181 tablet_bootstrap.cc:492] T bdaec169cfb5471391fcd3050bb00431 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:24.074028 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_7e88ecc0aa5e43ffb84af9b8dc6da938" } modify_external_catalogs: true
W20250112 02:52:24.074486 511 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 8cef1a66738c4bb3bad7019ce64f2ec2 because the tablet was not found. No further retry: Not found: Tablet not found: 8cef1a66738c4bb3bad7019ce64f2ec2
W20250112 02:52:24.074752 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 235e9dd0964c4ea6af7d05ef58acae0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 235e9dd0964c4ea6af7d05ef58acae0d already in progress: creating tablet
I20250112 02:52:24.075564 1181 tablet_bootstrap.cc:654] T bdaec169cfb5471391fcd3050bb00431 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:24.075902 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet f05d3326545849b9af4519bc66c21069 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f05d3326545849b9af4519bc66c21069 already in progress: creating tablet
W20250112 02:52:24.076097 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 09886cd08c714f24a96e4df02bc0e6c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09886cd08c714f24a96e4df02bc0e6c8 already in progress: creating tablet
I20250112 02:52:24.076162 1175 tablet_replica.cc:331] T 9c9a803ff19645328107d3b2fabb90c6 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
W20250112 02:52:24.076232 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 4bc067e895cb4c69af4dedc03533a276 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4bc067e895cb4c69af4dedc03533a276 already in progress: creating tablet
I20250112 02:52:24.076332 1175 raft_consensus.cc:2238] T 9c9a803ff19645328107d3b2fabb90c6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
W20250112 02:52:24.076366 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 28abc02b1b21427eae06560b99bcc078 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28abc02b1b21427eae06560b99bcc078 already in progress: creating tablet
I20250112 02:52:24.076522 1175 raft_consensus.cc:2267] T 9c9a803ff19645328107d3b2fabb90c6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.076740 1173 ts_tablet_manager.cc:1428] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.036s user 0.001s sys 0.002s
I20250112 02:52:24.078034 1181 tablet_bootstrap.cc:492] T bdaec169cfb5471391fcd3050bb00431 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:24.078163 1181 ts_tablet_manager.cc:1397] T bdaec169cfb5471391fcd3050bb00431 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20250112 02:52:24.078773 1181 raft_consensus.cc:357] T bdaec169cfb5471391fcd3050bb00431 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.078951 1181 raft_consensus.cc:383] T bdaec169cfb5471391fcd3050bb00431 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.079005 1181 raft_consensus.cc:738] T bdaec169cfb5471391fcd3050bb00431 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.079187 1181 consensus_queue.cc:260] T bdaec169cfb5471391fcd3050bb00431 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.080315 517 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3f5b418c17084420b867498fda79fb0a (table default.Table_0b78c0ecb1f245838a0117009873c0e9 [id=ad0be37e6b6f4b9597dcb4cbd50d5b12]). Expected version 1 got 0
I20250112 02:52:24.079676 1181 ts_tablet_manager.cc:1428] T bdaec169cfb5471391fcd3050bb00431 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250112 02:52:24.080852 1175 ts_tablet_manager.cc:1905] T 9c9a803ff19645328107d3b2fabb90c6 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.081907 1181 tablet_bootstrap.cc:492] T 2b34fb96bc5140118725af8346ea827f P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:24.083719 1181 tablet_bootstrap.cc:654] T 2b34fb96bc5140118725af8346ea827f P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.084900 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_126d7ad084814c798207d8f2938eebb8" } new_table_name: "default.Table_830855385a78429f908678179b435e11" modify_external_catalogs: true
I20250112 02:52:24.087906 1175 ts_tablet_manager.cc:1918] T 9c9a803ff19645328107d3b2fabb90c6 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.088220 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_62be2ee0d4744f198098621bf776cf01" } modify_external_catalogs: true
I20250112 02:52:24.088409 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_62be2ee0d4744f198098621bf776cf01" } modify_external_catalogs: true
I20250112 02:52:24.088472 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_b2950e148c2444f9b35e9f8eadcaf702"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:24.088650 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_1ace3ace3c994601a8102bd18c847fa5" } new_table_name: "default.Table_1b88ed105b4f49cd98fcf25354f71843" modify_external_catalogs: true
W20250112 02:52:24.089015 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2950e148c2444f9b35e9f8eadcaf702 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.089215 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_fb7e727aeed44203b2bb7e90a844e44b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:24.092866 1175 log.cc:1198] T 9c9a803ff19645328107d3b2fabb90c6 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/9c9a803ff19645328107d3b2fabb90c6
I20250112 02:52:24.093264 1175 ts_tablet_manager.cc:1939] T 9c9a803ff19645328107d3b2fabb90c6 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
W20250112 02:52:24.094012 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb7e727aeed44203b2bb7e90a844e44b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.095310 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_12a25f7a620047b69f34be9eeae64dd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.095817 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12a25f7a620047b69f34be9eeae64dd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.097584 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_a60b0bb4ae784e09bb62235bfb1d7bcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.098124 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a60b0bb4ae784e09bb62235bfb1d7bcd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:24.098601 525 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5601a9398c9a466582a98967c0e749e2 consensus state has no leader
I20250112 02:52:24.099893 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 9c9a803ff19645328107d3b2fabb90c6 (table default.Table_62be2ee0d4744f198098621bf776cf01 [id=bf2df21352c64c069c2a06c878911016]) successfully deleted
W20250112 02:52:24.100167 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet e8220011788a480eb50ee4b670c45739 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8220011788a480eb50ee4b670c45739 already in progress: creating tablet
W20250112 02:52:24.100409 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 328e307a4b35445f92cbfab24f2ada48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 328e307a4b35445f92cbfab24f2ada48 already in progress: creating tablet
I20250112 02:52:24.101030 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 11afb91c4fde4f6c8204c1cae22cffc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.101598 1175 tablet_replica.cc:331] T 11afb91c4fde4f6c8204c1cae22cffc5 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:24.101778 1175 raft_consensus.cc:2238] T 11afb91c4fde4f6c8204c1cae22cffc5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.101976 896 tablet_service.cc:1514] Processing DeleteTablet for tablet bdaec169cfb5471391fcd3050bb00431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.102144 1175 raft_consensus.cc:2267] T 11afb91c4fde4f6c8204c1cae22cffc5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.102387 1173 tablet_bootstrap.cc:492] T d02caa04543d43e086f5cd56ac1ead1a P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:24.102857 1175 ts_tablet_manager.cc:1905] T 11afb91c4fde4f6c8204c1cae22cffc5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.105221 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_c360a95859e5448f9af5f90629a898e4" } modify_external_catalogs: true
I20250112 02:52:24.105417 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_c360a95859e5448f9af5f90629a898e4" } modify_external_catalogs: true
I20250112 02:52:24.109771 753 tablet_service.cc:1467] Processing CreateTablet for tablet b885247d2cb449e0a29f6f5d81518589 (DEFAULT_TABLE table=default.Table_edaa8b50dfa8463e99e05ab8d1d89bc3 [id=a6a18be9a78e4959b7d272c447f6fc25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.110141 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b885247d2cb449e0a29f6f5d81518589. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.113550 897 tablet_service.cc:1467] Processing CreateTablet for tablet f6803d0f553b4204b415b2734286bf4a (DEFAULT_TABLE table=default.Table_c9cd1883fedc45e89202fd92fbbd0e50 [id=944bc8032e5b4d32a6d568676f53a401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.113961 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6803d0f553b4204b415b2734286bf4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.113651 1175 ts_tablet_manager.cc:1918] T 11afb91c4fde4f6c8204c1cae22cffc5 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.114149 1175 log.cc:1198] T 11afb91c4fde4f6c8204c1cae22cffc5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/11afb91c4fde4f6c8204c1cae22cffc5
I20250112 02:52:24.114673 1181 tablet_bootstrap.cc:492] T 2b34fb96bc5140118725af8346ea827f P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:24.114800 1181 ts_tablet_manager.cc:1397] T 2b34fb96bc5140118725af8346ea827f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.018s
I20250112 02:52:24.115582 1181 raft_consensus.cc:357] T 2b34fb96bc5140118725af8346ea827f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.115773 1181 raft_consensus.cc:383] T 2b34fb96bc5140118725af8346ea827f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.115864 1181 raft_consensus.cc:738] T 2b34fb96bc5140118725af8346ea827f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.116078 1181 consensus_queue.cc:260] T 2b34fb96bc5140118725af8346ea827f P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.116580 1181 ts_tablet_manager.cc:1428] T 2b34fb96bc5140118725af8346ea827f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:24.116832 1181 tablet_bootstrap.cc:492] T 22750265b522471680835e7f98475e6e P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:24.117218 760 tablet_service.cc:1467] Processing CreateTablet for tablet f6803d0f553b4204b415b2734286bf4a (DEFAULT_TABLE table=default.Table_c9cd1883fedc45e89202fd92fbbd0e50 [id=944bc8032e5b4d32a6d568676f53a401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.117512 1175 ts_tablet_manager.cc:1939] T 11afb91c4fde4f6c8204c1cae22cffc5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:24.118495 1175 tablet_replica.cc:331] T bdaec169cfb5471391fcd3050bb00431 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:24.118544 1181 tablet_bootstrap.cc:654] T 22750265b522471680835e7f98475e6e P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.118708 1175 raft_consensus.cc:2238] T bdaec169cfb5471391fcd3050bb00431 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.118863 1175 raft_consensus.cc:2267] T bdaec169cfb5471391fcd3050bb00431 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.119817 896 tablet_service.cc:1467] Processing CreateTablet for tablet a1a0a13696fc41ccb28218f1275a4b5d (DEFAULT_TABLE table=default.Table_62be2ee0d4744f198098621bf776cf01 [id=bf2df21352c64c069c2a06c878911016]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.120232 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1a0a13696fc41ccb28218f1275a4b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.120208 750 tablet_service.cc:1467] Processing CreateTablet for tablet a1a0a13696fc41ccb28218f1275a4b5d (DEFAULT_TABLE table=default.Table_62be2ee0d4744f198098621bf776cf01 [id=bf2df21352c64c069c2a06c878911016]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.123320 1178 raft_consensus.cc:491] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:24.123809 893 tablet_service.cc:1467] Processing CreateTablet for tablet b885247d2cb449e0a29f6f5d81518589 (DEFAULT_TABLE table=default.Table_edaa8b50dfa8463e99e05ab8d1d89bc3 [id=a6a18be9a78e4959b7d272c447f6fc25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.124210 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b885247d2cb449e0a29f6f5d81518589. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.123493 1023 tablet_service.cc:1467] Processing CreateTablet for tablet a47cdc208fbc431b8326364e84432cf5 (DEFAULT_TABLE table=default.Table_bd1cdf8c9b3343af8d17a9fd05c77145 [id=c19f442c5dc7459389b6bac8beda390c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.126480 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a47cdc208fbc431b8326364e84432cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.126555 1181 tablet_bootstrap.cc:492] T 22750265b522471680835e7f98475e6e P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:24.126704 1181 ts_tablet_manager.cc:1397] T 22750265b522471680835e7f98475e6e P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20250112 02:52:24.127408 1181 raft_consensus.cc:357] T 22750265b522471680835e7f98475e6e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.127655 1181 raft_consensus.cc:383] T 22750265b522471680835e7f98475e6e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.127737 1181 raft_consensus.cc:738] T 22750265b522471680835e7f98475e6e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.127990 1181 consensus_queue.cc:260] T 22750265b522471680835e7f98475e6e P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.128439 1178 raft_consensus.cc:513] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.128686 1181 ts_tablet_manager.cc:1428] T 22750265b522471680835e7f98475e6e P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:24.128865 1175 ts_tablet_manager.cc:1905] T bdaec169cfb5471391fcd3050bb00431 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.129007 1181 tablet_bootstrap.cc:492] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:24.129125 1178 leader_election.cc:290] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:24.130246 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 11afb91c4fde4f6c8204c1cae22cffc5 (table default.Table_a78a69e0b1914d25a0f0726d75b66731 [id=e47433a989db4945973f25468fd1581b]) successfully deleted
I20250112 02:52:24.130710 1181 tablet_bootstrap.cc:654] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.133649 1181 tablet_bootstrap.cc:492] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:24.133787 1181 ts_tablet_manager.cc:1397] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250112 02:52:24.134479 1181 raft_consensus.cc:357] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.134704 1181 raft_consensus.cc:383] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.134779 1181 raft_consensus.cc:738] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.134994 1181 consensus_queue.cc:260] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.135658 1181 ts_tablet_manager.cc:1428] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:24.135870 1181 tablet_bootstrap.cc:492] T 6834be02864543469056a928fafe28ca P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
W20250112 02:52:24.136319 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 71b5f80ed198491780cbe17e6cd3fa15 consensus state has no leader
I20250112 02:52:24.137315 1181 tablet_bootstrap.cc:654] T 6834be02864543469056a928fafe28ca P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.137347 1022 tablet_service.cc:1467] Processing CreateTablet for tablet f6803d0f553b4204b415b2734286bf4a (DEFAULT_TABLE table=default.Table_c9cd1883fedc45e89202fd92fbbd0e50 [id=944bc8032e5b4d32a6d568676f53a401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.138211 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6803d0f553b4204b415b2734286bf4a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:24.139724 989 leader_election.cc:343] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 9c9a803ff19645328107d3b2fabb90c6
I20250112 02:52:24.141558 1175 ts_tablet_manager.cc:1918] T bdaec169cfb5471391fcd3050bb00431 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.141685 1175 log.cc:1198] T bdaec169cfb5471391fcd3050bb00431 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/bdaec169cfb5471391fcd3050bb00431
I20250112 02:52:24.142004 1175 ts_tablet_manager.cc:1939] T bdaec169cfb5471391fcd3050bb00431 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:24.143054 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet bdaec169cfb5471391fcd3050bb00431 (table default.Table_7d31752a7e7246ad9806310039f73c14 [id=62176e03ce0b4ced96e5083fa32b1826]) successfully deleted
I20250112 02:52:24.144727 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_c2d18d092b5a46f8b59f64aa4a8f02c7" } modify_external_catalogs: true
I20250112 02:52:24.144893 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_c2d18d092b5a46f8b59f64aa4a8f02c7" } modify_external_catalogs: true
I20250112 02:52:24.145529 1177 tablet_bootstrap.cc:492] T 11afb91c4fde4f6c8204c1cae22cffc5 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:24.145673 1177 ts_tablet_manager.cc:1397] T 11afb91c4fde4f6c8204c1cae22cffc5 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.000s
I20250112 02:52:24.146402 1177 raft_consensus.cc:357] T 11afb91c4fde4f6c8204c1cae22cffc5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.146605 1177 raft_consensus.cc:383] T 11afb91c4fde4f6c8204c1cae22cffc5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.146704 1177 raft_consensus.cc:738] T 11afb91c4fde4f6c8204c1cae22cffc5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.146893 1177 consensus_queue.cc:260] T 11afb91c4fde4f6c8204c1cae22cffc5 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.147390 1177 ts_tablet_manager.cc:1428] T 11afb91c4fde4f6c8204c1cae22cffc5 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:24.149997 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet aa67bf25dc894d7ea96d247a22c14e03
I20250112 02:52:24.150723 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet a1a0a13696fc41ccb28218f1275a4b5d
I20250112 02:52:24.150555 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_e21cbee8871f4ab583b22a3eaa4b5e6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:24.150967 759 tablet_service.cc:1514] Processing DeleteTablet for tablet aa67bf25dc894d7ea96d247a22c14e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:24.151175 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e21cbee8871f4ab583b22a3eaa4b5e6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.151582 759 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a0a13696fc41ccb28218f1275a4b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.152581 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_060115627d5f4bd786853027f080f260"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.152968 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_060115627d5f4bd786853027f080f260 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.154058 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7
I20250112 02:52:24.154274 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_47f5f06ece8945aba6226a9c4c928e66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.154675 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47f5f06ece8945aba6226a9c4c928e66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.155052 759 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.155418 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_47effe293cfd4ebfa64ad81407d7fe93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:24.155730 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_1bef29fabb1d4271b04c6c5c3f7c89d5" } new_table_name: "default.Table_e8041534ae864796a75a6edf2e4a33dc" modify_external_catalogs: true
W20250112 02:52:24.155952 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47effe293cfd4ebfa64ad81407d7fe93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.163793 899 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.164655 897 tablet_service.cc:1514] Processing DeleteTablet for tablet aa67bf25dc894d7ea96d247a22c14e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:24.165033 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 already in progress: creating tablet
I20250112 02:52:24.165335 893 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a0a13696fc41ccb28218f1275a4b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.165583 896 tablet_service.cc:1514] Processing DeleteTablet for tablet bdaec169cfb5471391fcd3050bb00431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:24.166266 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet aa67bf25dc894d7ea96d247a22c14e03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa67bf25dc894d7ea96d247a22c14e03 already in progress: creating tablet
W20250112 02:52:24.166494 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet a1a0a13696fc41ccb28218f1275a4b5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1a0a13696fc41ccb28218f1275a4b5d already in progress: creating tablet
W20250112 02:52:24.166741 511 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet bdaec169cfb5471391fcd3050bb00431 because the tablet was not found. No further retry: Not found: Tablet not found: bdaec169cfb5471391fcd3050bb00431
I20250112 02:52:24.159607 1021 tablet_service.cc:1467] Processing CreateTablet for tablet b885247d2cb449e0a29f6f5d81518589 (DEFAULT_TABLE table=default.Table_edaa8b50dfa8463e99e05ab8d1d89bc3 [id=a6a18be9a78e4959b7d272c447f6fc25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.169925 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b885247d2cb449e0a29f6f5d81518589. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.171231 1028 tablet_service.cc:1467] Processing CreateTablet for tablet a1a0a13696fc41ccb28218f1275a4b5d (DEFAULT_TABLE table=default.Table_62be2ee0d4744f198098621bf776cf01 [id=bf2df21352c64c069c2a06c878911016]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.172310 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a0a13696fc41ccb28218f1275a4b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.173962 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1a0a13696fc41ccb28218f1275a4b5d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:24.177364 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 consensus state has no leader
I20250112 02:52:24.178058 759 tablet_service.cc:1467] Processing CreateTablet for tablet 6b521522c4784d2ab4056268823f1099 (DEFAULT_TABLE table=default.Table_a60b0bb4ae784e09bb62235bfb1d7bcd [id=f8cd4289f89a4a748c26a7a01236406e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.179107 756 tablet_service.cc:1467] Processing CreateTablet for tablet aba1744138fe42ae9d097db01cac083d (DEFAULT_TABLE table=default.Table_fb7e727aeed44203b2bb7e90a844e44b [id=587926735ced4c17b747f5c1fbee9f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.179605 896 tablet_service.cc:1467] Processing CreateTablet for tablet 6b521522c4784d2ab4056268823f1099 (DEFAULT_TABLE table=default.Table_a60b0bb4ae784e09bb62235bfb1d7bcd [id=f8cd4289f89a4a748c26a7a01236406e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.180012 893 tablet_service.cc:1467] Processing CreateTablet for tablet aba1744138fe42ae9d097db01cac083d (DEFAULT_TABLE table=default.Table_fb7e727aeed44203b2bb7e90a844e44b [id=587926735ced4c17b747f5c1fbee9f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.180096 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b521522c4784d2ab4056268823f1099. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.180557 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aba1744138fe42ae9d097db01cac083d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.181022 1181 tablet_bootstrap.cc:492] T 6834be02864543469056a928fafe28ca P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:24.181211 1181 ts_tablet_manager.cc:1397] T 6834be02864543469056a928fafe28ca P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.009s
I20250112 02:52:24.182744 757 tablet_service.cc:1467] Processing CreateTablet for tablet cb724feccc7f4fbc846fb6fa61be3385 (DEFAULT_TABLE table=default.Table_b2950e148c2444f9b35e9f8eadcaf702 [id=81a09e598ee34508aa88abfc6b44a2bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.182801 1181 raft_consensus.cc:357] T 6834be02864543469056a928fafe28ca P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.183411 1181 raft_consensus.cc:383] T 6834be02864543469056a928fafe28ca P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.183593 1181 raft_consensus.cc:738] T 6834be02864543469056a928fafe28ca P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.183832 1181 consensus_queue.cc:260] T 6834be02864543469056a928fafe28ca P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.184347 1181 ts_tablet_manager.cc:1428] T 6834be02864543469056a928fafe28ca P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250112 02:52:24.184649 1181 tablet_bootstrap.cc:492] T 2728a022ef0d4952966d397a32356503 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:24.186319 1181 tablet_bootstrap.cc:654] T 2728a022ef0d4952966d397a32356503 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.188115 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.188516 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 2b34fb96bc5140118725af8346ea827f
I20250112 02:52:24.189736 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b34fb96bc5140118725af8346ea827f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.190009 1175 tablet_replica.cc:331] T 2b34fb96bc5140118725af8346ea827f P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:24.190181 1175 raft_consensus.cc:2238] T 2b34fb96bc5140118725af8346ea827f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.190358 1175 raft_consensus.cc:2267] T 2b34fb96bc5140118725af8346ea827f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.190560 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b34fb96bc5140118725af8346ea827f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.191331 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b34fb96bc5140118725af8346ea827f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.191344 1175 ts_tablet_manager.cc:1905] T 2b34fb96bc5140118725af8346ea827f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.192116 1034 tablet_service.cc:1467] Processing CreateTablet for tablet aba1744138fe42ae9d097db01cac083d (DEFAULT_TABLE table=default.Table_fb7e727aeed44203b2bb7e90a844e44b [id=587926735ced4c17b747f5c1fbee9f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.195992 1181 tablet_bootstrap.cc:492] T 2728a022ef0d4952966d397a32356503 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:24.196131 1181 ts_tablet_manager.cc:1397] T 2728a022ef0d4952966d397a32356503 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20250112 02:52:24.196240 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aba1744138fe42ae9d097db01cac083d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.196919 1181 raft_consensus.cc:357] T 2728a022ef0d4952966d397a32356503 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.197122 1181 raft_consensus.cc:383] T 2728a022ef0d4952966d397a32356503 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.197227 1181 raft_consensus.cc:738] T 2728a022ef0d4952966d397a32356503 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.197449 1181 consensus_queue.cc:260] T 2728a022ef0d4952966d397a32356503 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.197935 1181 ts_tablet_manager.cc:1428] T 2728a022ef0d4952966d397a32356503 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:24.198171 1181 tablet_bootstrap.cc:492] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:24.199090 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 22750265b522471680835e7f98475e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.199854 1181 tablet_bootstrap.cc:654] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.200208 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet aa67bf25dc894d7ea96d247a22c14e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.202061 525 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 2728a022ef0d4952966d397a32356503 (table default.Table_6d358fc697c74529822f0b2b70321a02 [id=9b36703d61a046799cb709c7618b1fbd]). Expected version 1 got 0
I20250112 02:52:24.203336 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 6834be02864543469056a928fafe28ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.203647 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b34fb96bc5140118725af8346ea827f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.204603 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 6b521522c4784d2ab4056268823f1099 (DEFAULT_TABLE table=default.Table_a60b0bb4ae784e09bb62235bfb1d7bcd [id=f8cd4289f89a4a748c26a7a01236406e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.204939 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b521522c4784d2ab4056268823f1099. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.205276 1175 ts_tablet_manager.cc:1918] T 2b34fb96bc5140118725af8346ea827f P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.205421 1175 log.cc:1198] T 2b34fb96bc5140118725af8346ea827f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/2b34fb96bc5140118725af8346ea827f
I20250112 02:52:24.205910 1175 ts_tablet_manager.cc:1939] T 2b34fb96bc5140118725af8346ea827f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:24.206990 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 2b34fb96bc5140118725af8346ea827f (table default.Table_c2d18d092b5a46f8b59f64aa4a8f02c7 [id=059958c4e1c74d8d8e4c029030d85895]) successfully deleted
I20250112 02:52:24.207724 1181 tablet_bootstrap.cc:492] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:24.207877 1181 ts_tablet_manager.cc:1397] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
W20250112 02:52:24.208295 511 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 2b34fb96bc5140118725af8346ea827f because the tablet was not found. No further retry: Not found: Tablet not found: 2b34fb96bc5140118725af8346ea827f
I20250112 02:52:24.208606 1181 raft_consensus.cc:357] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:24.208812 1181 raft_consensus.cc:383] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.208904 1181 raft_consensus.cc:738] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.207875 897 tablet_service.cc:1467] Processing CreateTablet for tablet cb724feccc7f4fbc846fb6fa61be3385 (DEFAULT_TABLE table=default.Table_b2950e148c2444f9b35e9f8eadcaf702 [id=81a09e598ee34508aa88abfc6b44a2bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.209311 751 tablet_service.cc:1467] Processing CreateTablet for tablet b2d8ea2ef18a46508371b4789ab6f4fa (DEFAULT_TABLE table=default.Table_12a25f7a620047b69f34be9eeae64dd2 [id=87c1bff49b4346599bf98166ce8672d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.209417 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb724feccc7f4fbc846fb6fa61be3385. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.209687 1175 tablet_replica.cc:331] T 22750265b522471680835e7f98475e6e P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:24.210335 1175 raft_consensus.cc:2238] T 22750265b522471680835e7f98475e6e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.210762 1175 raft_consensus.cc:2267] T 22750265b522471680835e7f98475e6e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.211112 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_f7982d4045f04157ae3aa0e8e8f6ffda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:24.211474 893 tablet_service.cc:1467] Processing CreateTablet for tablet b2d8ea2ef18a46508371b4789ab6f4fa (DEFAULT_TABLE table=default.Table_12a25f7a620047b69f34be9eeae64dd2 [id=87c1bff49b4346599bf98166ce8672d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.211853 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2d8ea2ef18a46508371b4789ab6f4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.214107 1175 ts_tablet_manager.cc:1905] T 22750265b522471680835e7f98475e6e P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.216964 1173 tablet_bootstrap.cc:654] T d02caa04543d43e086f5cd56ac1ead1a P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.218978 1030 tablet_service.cc:1467] Processing CreateTablet for tablet b2d8ea2ef18a46508371b4789ab6f4fa (DEFAULT_TABLE table=default.Table_12a25f7a620047b69f34be9eeae64dd2 [id=87c1bff49b4346599bf98166ce8672d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.219303 1181 consensus_queue.cc:260] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:24.219942 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7982d4045f04157ae3aa0e8e8f6ffda in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.222064 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2d8ea2ef18a46508371b4789ab6f4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.222872 1181 ts_tablet_manager.cc:1428] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20250112 02:52:24.223176 1181 tablet_bootstrap.cc:492] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:24.227190 1175 ts_tablet_manager.cc:1918] T 22750265b522471680835e7f98475e6e P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.227319 1175 log.cc:1198] T 22750265b522471680835e7f98475e6e P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/22750265b522471680835e7f98475e6e
I20250112 02:52:24.231257 754 tablet_service.cc:1467] Processing CreateTablet for tablet e9fc9a4f1e8944abb76dad6c2ad53d35 (DEFAULT_TABLE table=default.Table_47effe293cfd4ebfa64ad81407d7fe93 [id=268dae304c9143a881c7a0898632a3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.232704 893 tablet_service.cc:1467] Processing CreateTablet for tablet e9fc9a4f1e8944abb76dad6c2ad53d35 (DEFAULT_TABLE table=default.Table_47effe293cfd4ebfa64ad81407d7fe93 [id=268dae304c9143a881c7a0898632a3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.233108 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9fc9a4f1e8944abb76dad6c2ad53d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.233081 758 tablet_service.cc:1467] Processing CreateTablet for tablet 09d06e2e8c13464db8cf801a292bdc99 (DEFAULT_TABLE table=default.Table_47f5f06ece8945aba6226a9c4c928e66 [id=e3bb489c53454222a79702b798c0c2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.234125 899 tablet_service.cc:1467] Processing CreateTablet for tablet 09d06e2e8c13464db8cf801a292bdc99 (DEFAULT_TABLE table=default.Table_47f5f06ece8945aba6226a9c4c928e66 [id=e3bb489c53454222a79702b798c0c2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.234472 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09d06e2e8c13464db8cf801a292bdc99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.234557 1031 tablet_service.cc:1467] Processing CreateTablet for tablet cb724feccc7f4fbc846fb6fa61be3385 (DEFAULT_TABLE table=default.Table_b2950e148c2444f9b35e9f8eadcaf702 [id=81a09e598ee34508aa88abfc6b44a2bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.228044 519 master_service.cc:946] ReplaceTablet: received request to replace tablet d02caa04543d43e086f5cd56ac1ead1a from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:24.230201 524 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5a8e1b96accd459eaed221ded4a453b6 (table default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94]). Expected version 2 got 0
I20250112 02:52:24.237082 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_79dbb2fcd6fe4c939bc0637b4ddf1b1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.238629 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79dbb2fcd6fe4c939bc0637b4ddf1b1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.240952 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6803d0f553b4204b415b2734286bf4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.241279 519 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5a8e1b96accd459eaed221ded4a453b6 (table default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94]). Expected version 2 got 0
I20250112 02:52:24.243804 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09d06e2e8c13464db8cf801a292bdc99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.244704 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_fb7e727aeed44203b2bb7e90a844e44b" } modify_external_catalogs: true
I20250112 02:52:24.244875 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_fb7e727aeed44203b2bb7e90a844e44b" } modify_external_catalogs: true
W20250112 02:52:24.245263 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet aa67bf25dc894d7ea96d247a22c14e03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa67bf25dc894d7ea96d247a22c14e03 already in progress: creating tablet
W20250112 02:52:24.245502 513 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet a1a0a13696fc41ccb28218f1275a4b5d because the tablet was not found. No further retry: Not found: Tablet not found: a1a0a13696fc41ccb28218f1275a4b5d
W20250112 02:52:24.245721 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 already in progress: creating tablet
W20250112 02:52:24.245950 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 2b34fb96bc5140118725af8346ea827f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b34fb96bc5140118725af8346ea827f already in progress: creating tablet
W20250112 02:52:24.246634 991 leader_election.cc:343] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: 9c9a803ff19645328107d3b2fabb90c6
I20250112 02:52:24.248060 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aba1744138fe42ae9d097db01cac083d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.250603 524 master_service.cc:946] ReplaceTablet: received request to replace tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:24.251678 991 leader_election.cc:304] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:24.251848 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb724feccc7f4fbc846fb6fa61be3385. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.252125 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b521522c4784d2ab4056268823f1099. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.252310 1178 raft_consensus.cc:2743] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:24.252549 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 09d06e2e8c13464db8cf801a292bdc99 (DEFAULT_TABLE table=default.Table_47f5f06ece8945aba6226a9c4c928e66 [id=e3bb489c53454222a79702b798c0c2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.255185 1172 ts_tablet_manager.cc:1918] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.255370 1172 log.cc:1198] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/1ed69c15f3d1448b929c3e9c768deb1f
I20250112 02:52:24.255596 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2d8ea2ef18a46508371b4789ab6f4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.256310 1172 ts_tablet_manager.cc:1939] T 1ed69c15f3d1448b929c3e9c768deb1f P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:24.257016 1020 tablet_service.cc:1467] Processing CreateTablet for tablet e9fc9a4f1e8944abb76dad6c2ad53d35 (DEFAULT_TABLE table=default.Table_47effe293cfd4ebfa64ad81407d7fe93 [id=268dae304c9143a881c7a0898632a3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.258455 1172 tablet_replica.cc:331] T d9125dd3f9544a54ac373ae0f90cce5f P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:24.261603 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 1ed69c15f3d1448b929c3e9c768deb1f (table default.Table_1a4e5b8794744193902634c51c0285be [id=f20d22fe09e94ecb8dbe6d4f0a4a8b28]) successfully deleted
W20250112 02:52:24.261826 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 09886cd08c714f24a96e4df02bc0e6c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09886cd08c714f24a96e4df02bc0e6c8 already in progress: creating tablet
W20250112 02:52:24.261999 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f05d3326545849b9af4519bc66c21069 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f05d3326545849b9af4519bc66c21069 already in progress: creating tablet
W20250112 02:52:24.262226 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 235e9dd0964c4ea6af7d05ef58acae0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 235e9dd0964c4ea6af7d05ef58acae0d already in progress: creating tablet
I20250112 02:52:24.262773 1177 tablet_bootstrap.cc:492] T 2b34fb96bc5140118725af8346ea827f P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:24.264114 1177 tablet_bootstrap.cc:654] T 2b34fb96bc5140118725af8346ea827f P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.265416 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1a0a13696fc41ccb28218f1275a4b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.266073 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9fc9a4f1e8944abb76dad6c2ad53d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.279268 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet aba1744138fe42ae9d097db01cac083d
I20250112 02:52:24.280747 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_393b6f264860440ebac3e3148390e317" } new_table_name: "default.Table_d462fac16720400f879bf69c1cdbcd6d" modify_external_catalogs: true
I20250112 02:52:24.280779 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_c8b1e93e51184d488f851b546d2aaaf7" } new_table_name: "default.Table_57a7c003d55441cc9b836feecff7ca2f" modify_external_catalogs: true
I20250112 02:52:24.281258 899 tablet_service.cc:1514] Processing DeleteTablet for tablet aba1744138fe42ae9d097db01cac083d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.283149 1181 tablet_bootstrap.cc:654] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:24.287734 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 28abc02b1b21427eae06560b99bcc078 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28abc02b1b21427eae06560b99bcc078 already in progress: creating tablet
W20250112 02:52:24.288022 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 4bc067e895cb4c69af4dedc03533a276 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4bc067e895cb4c69af4dedc03533a276 already in progress: creating tablet
I20250112 02:52:24.289384 1175 ts_tablet_manager.cc:1939] T 22750265b522471680835e7f98475e6e P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
W20250112 02:52:24.289520 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 97b31ebe035c4552baaffc3133ec73c1 consensus state has no leader
I20250112 02:52:24.290341 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 22750265b522471680835e7f98475e6e (table default.Table_89761f48fc8547e8b786c9559485f243 [id=c8799d47318d42d9859e36a37a5a9445]) successfully deleted
I20250112 02:52:24.291640 899 tablet_service.cc:1467] Processing CreateTablet for tablet 1726d61cc36a4deb83c5d604e8daf4f0 (DEFAULT_TABLE table=default.Table_060115627d5f4bd786853027f080f260 [id=34b5449c89e248189460592f6eae98c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.292029 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1726d61cc36a4deb83c5d604e8daf4f0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:24.293181 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6dc79f4d1194f81963a8094ed6b0d2e consensus state has no leader
I20250112 02:52:24.295502 759 tablet_service.cc:1514] Processing DeleteTablet for tablet aba1744138fe42ae9d097db01cac083d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:24.296356 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet aba1744138fe42ae9d097db01cac083d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aba1744138fe42ae9d097db01cac083d already in progress: creating tablet
I20250112 02:52:24.296450 893 tablet_service.cc:1467] Processing CreateTablet for tablet afa2f8aabe5e42ccbc15b193fdc03563 (DEFAULT_TABLE table=default.Table_e21cbee8871f4ab583b22a3eaa4b5e6a [id=1b3ac0e8c0704c618077010dda4f7413]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.296752 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afa2f8aabe5e42ccbc15b193fdc03563. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.297454 754 tablet_service.cc:1467] Processing CreateTablet for tablet afa2f8aabe5e42ccbc15b193fdc03563 (DEFAULT_TABLE table=default.Table_e21cbee8871f4ab583b22a3eaa4b5e6a [id=1b3ac0e8c0704c618077010dda4f7413]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.297792 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 11afb91c4fde4f6c8204c1cae22cffc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.297991 1171 tablet_replica.cc:331] T 11afb91c4fde4f6c8204c1cae22cffc5 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:24.298079 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet aba1744138fe42ae9d097db01cac083d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.298135 1171 raft_consensus.cc:2238] T 11afb91c4fde4f6c8204c1cae22cffc5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.298285 1171 raft_consensus.cc:2267] T 11afb91c4fde4f6c8204c1cae22cffc5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.298844 1171 ts_tablet_manager.cc:1905] T 11afb91c4fde4f6c8204c1cae22cffc5 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.299075 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afa2f8aabe5e42ccbc15b193fdc03563. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.304265 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_56ea07abda424090a7942efabc871092"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.304742 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56ea07abda424090a7942efabc871092 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.305936 1175 tablet_replica.cc:331] T 6834be02864543469056a928fafe28ca P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:24.306118 1175 raft_consensus.cc:2238] T 6834be02864543469056a928fafe28ca P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.306289 1175 raft_consensus.cc:2267] T 6834be02864543469056a928fafe28ca P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.307593 1034 tablet_service.cc:1467] Processing CreateTablet for tablet afa2f8aabe5e42ccbc15b193fdc03563 (DEFAULT_TABLE table=default.Table_e21cbee8871f4ab583b22a3eaa4b5e6a [id=1b3ac0e8c0704c618077010dda4f7413]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.308135 760 tablet_service.cc:1467] Processing CreateTablet for tablet 1726d61cc36a4deb83c5d604e8daf4f0 (DEFAULT_TABLE table=default.Table_060115627d5f4bd786853027f080f260 [id=34b5449c89e248189460592f6eae98c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.309023 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_b2950e148c2444f9b35e9f8eadcaf702" } new_table_name: "default.Table_24713b20c8e044f58b27831ee0f2580c" modify_external_catalogs: true
I20250112 02:52:24.309432 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1726d61cc36a4deb83c5d604e8daf4f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.311506 1175 ts_tablet_manager.cc:1905] T 6834be02864543469056a928fafe28ca P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.312016 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb724feccc7f4fbc846fb6fa61be3385. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.313588 1171 ts_tablet_manager.cc:1918] T 11afb91c4fde4f6c8204c1cae22cffc5 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.313989 1172 raft_consensus.cc:2238] T d9125dd3f9544a54ac373ae0f90cce5f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.314708 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09d06e2e8c13464db8cf801a292bdc99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.315183 1173 tablet_bootstrap.cc:492] T d02caa04543d43e086f5cd56ac1ead1a P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:24.316543 1171 log.cc:1198] T 11afb91c4fde4f6c8204c1cae22cffc5 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/11afb91c4fde4f6c8204c1cae22cffc5
I20250112 02:52:24.317413 1171 ts_tablet_manager.cc:1939] T 11afb91c4fde4f6c8204c1cae22cffc5 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:24.319582 1172 raft_consensus.cc:2267] T d9125dd3f9544a54ac373ae0f90cce5f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.322417 522 master_service.cc:946] ReplaceTablet: received request to replace tablet 2728a022ef0d4952966d397a32356503 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:24.323604 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 11afb91c4fde4f6c8204c1cae22cffc5 (table default.Table_a78a69e0b1914d25a0f0726d75b66731 [id=e47433a989db4945973f25468fd1581b]) successfully deleted
I20250112 02:52:24.324247 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9fc9a4f1e8944abb76dad6c2ad53d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.325843 1177 tablet_bootstrap.cc:492] T 2b34fb96bc5140118725af8346ea827f P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:24.325975 1177 ts_tablet_manager.cc:1397] T 2b34fb96bc5140118725af8346ea827f P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.063s user 0.001s sys 0.001s
I20250112 02:52:24.326596 1177 raft_consensus.cc:357] T 2b34fb96bc5140118725af8346ea827f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.326819 1177 raft_consensus.cc:383] T 2b34fb96bc5140118725af8346ea827f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.326869 1177 raft_consensus.cc:738] T 2b34fb96bc5140118725af8346ea827f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.327307 1177 consensus_queue.cc:260] T 2b34fb96bc5140118725af8346ea827f P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.331007 1177 ts_tablet_manager.cc:1428] T 2b34fb96bc5140118725af8346ea827f P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 02:52:24.332067 1173 ts_tablet_manager.cc:1397] T d02caa04543d43e086f5cd56ac1ead1a P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.238s user 0.000s sys 0.003s
I20250112 02:52:24.332491 1177 tablet_bootstrap.cc:492] T 22750265b522471680835e7f98475e6e P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:24.332676 1173 raft_consensus.cc:357] T d02caa04543d43e086f5cd56ac1ead1a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.332866 1173 raft_consensus.cc:383] T d02caa04543d43e086f5cd56ac1ead1a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.332935 1173 raft_consensus.cc:738] T d02caa04543d43e086f5cd56ac1ead1a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.333150 1173 consensus_queue.cc:260] T d02caa04543d43e086f5cd56ac1ead1a P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.334978 1177 tablet_bootstrap.cc:654] T 22750265b522471680835e7f98475e6e P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.335191 1173 ts_tablet_manager.cc:1428] T d02caa04543d43e086f5cd56ac1ead1a P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250112 02:52:24.335467 1173 tablet_bootstrap.cc:492] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:24.336088 525 master_service.cc:946] ReplaceTablet: received request to replace tablet 6b521522c4784d2ab4056268823f1099 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:24.336580 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_945067c243f84394957a328d65592441"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:24.336959 1173 tablet_bootstrap.cc:654] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:24.336997 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_945067c243f84394957a328d65592441 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.342561 524 master_service.cc:946] ReplaceTablet: received request to replace tablet d02caa04543d43e086f5cd56ac1ead1a from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:24.343137 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b34fb96bc5140118725af8346ea827f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.343369 1171 tablet_replica.cc:331] T 2b34fb96bc5140118725af8346ea827f P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:24.343593 1171 raft_consensus.cc:2238] T 2b34fb96bc5140118725af8346ea827f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.343844 1171 raft_consensus.cc:2267] T 2b34fb96bc5140118725af8346ea827f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:24.347296 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb724feccc7f4fbc846fb6fa61be3385 consensus state has no leader
I20250112 02:52:24.348083 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet d02caa04543d43e086f5cd56ac1ead1a with delete_type TABLET_DATA_DELETED (replaced by tablet 5a8e1b96accd459eaed221ded4a453b6) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.348695 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afa2f8aabe5e42ccbc15b193fdc03563. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.351236 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_d588e6b08cae48e590a4a57002df6323"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.351964 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d588e6b08cae48e590a4a57002df6323 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.352123 1172 ts_tablet_manager.cc:1905] T d9125dd3f9544a54ac373ae0f90cce5f P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.355916 1031 tablet_service.cc:1467] Processing CreateTablet for tablet ecfc7d8ba5e4406ebc61f37e6d500473 (DEFAULT_TABLE table=default.Table_79dbb2fcd6fe4c939bc0637b4ddf1b1e [id=44975f62b4f9429981075254868cb170]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.356216 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecfc7d8ba5e4406ebc61f37e6d500473. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.356673 899 tablet_service.cc:1467] Processing CreateTablet for tablet ecfc7d8ba5e4406ebc61f37e6d500473 (DEFAULT_TABLE table=default.Table_79dbb2fcd6fe4c939bc0637b4ddf1b1e [id=44975f62b4f9429981075254868cb170]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.357064 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecfc7d8ba5e4406ebc61f37e6d500473. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.362316 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_84c29ed08966455b9dde3dfcc832ffac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.362877 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84c29ed08966455b9dde3dfcc832ffac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.366253 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 6b521522c4784d2ab4056268823f1099
I20250112 02:52:24.366694 525 catalog_manager.cc:6596] ReplaceTablet: tablet 6b521522c4784d2ab4056268823f1099 deleted and replaced by tablet ab2e31a65294407bb62b13e92898a7d2
I20250112 02:52:24.369369 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_6d358fc697c74529822f0b2b70321a02" } modify_external_catalogs: true
I20250112 02:52:24.369585 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_6d358fc697c74529822f0b2b70321a02" } modify_external_catalogs: true
I20250112 02:52:24.369905 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 1726d61cc36a4deb83c5d604e8daf4f0 (DEFAULT_TABLE table=default.Table_060115627d5f4bd786853027f080f260 [id=34b5449c89e248189460592f6eae98c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.370270 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1726d61cc36a4deb83c5d604e8daf4f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.370708 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_01a0b2e48fcd4a769a3d442390622c1e" } modify_external_catalogs: true
I20250112 02:52:24.370925 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_01a0b2e48fcd4a769a3d442390622c1e" } modify_external_catalogs: true
I20250112 02:52:24.371186 893 tablet_service.cc:1467] Processing CreateTablet for tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 (DEFAULT_TABLE table=default.Table_f7982d4045f04157ae3aa0e8e8f6ffda [id=2f43ebd749304c0287769224ac6b12c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.371575 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.373047 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_1b88ed105b4f49cd98fcf25354f71843" } new_table_name: "default.Table_247a821819994de6adac6fabd06c3a1f" modify_external_catalogs: true
I20250112 02:52:24.374459 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b521522c4784d2ab4056268823f1099 with delete_type TABLET_DATA_DELETED (replaced by tablet ab2e31a65294407bb62b13e92898a7d2) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.375433 1171 ts_tablet_manager.cc:1905] T 2b34fb96bc5140118725af8346ea827f P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.375480 760 tablet_service.cc:1467] Processing CreateTablet for tablet ecfc7d8ba5e4406ebc61f37e6d500473 (DEFAULT_TABLE table=default.Table_79dbb2fcd6fe4c939bc0637b4ddf1b1e [id=44975f62b4f9429981075254868cb170]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.375761 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 (DEFAULT_TABLE table=default.Table_f7982d4045f04157ae3aa0e8e8f6ffda [id=2f43ebd749304c0287769224ac6b12c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.375917 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecfc7d8ba5e4406ebc61f37e6d500473. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.376102 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.380826 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 2728a022ef0d4952966d397a32356503
I20250112 02:52:24.381314 522 catalog_manager.cc:6596] ReplaceTablet: tablet 2728a022ef0d4952966d397a32356503 deleted and replaced by tablet bf58fb4538d946e1818a5d31d2a988e2
I20250112 02:52:24.382603 750 tablet_service.cc:1467] Processing CreateTablet for tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 (DEFAULT_TABLE table=default.Table_f7982d4045f04157ae3aa0e8e8f6ffda [id=2f43ebd749304c0287769224ac6b12c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.382977 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.385046 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_82662e02aee0459dad1b621df36f58f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:24.385170 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_672d7452ad6d4c99ae9f2ae77b592f78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.385581 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82662e02aee0459dad1b621df36f58f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.386687 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_e390b5d103344bf89b480cd408388b55" } new_table_name: "default.Table_1216acc1fa6b4e51a55a89ffe22bcc9a" modify_external_catalogs: true
I20250112 02:52:24.387594 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 2728a022ef0d4952966d397a32356503 with delete_type TABLET_DATA_DELETED (replaced by tablet bf58fb4538d946e1818a5d31d2a988e2) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.387950 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_893bf955f62b495c9dcc59390b472f96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.388363 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_893bf955f62b495c9dcc59390b472f96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.389402 1193 raft_consensus.cc:491] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:24.389535 1193 raft_consensus.cc:513] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.390237 1193 leader_election.cc:290] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:24.390975 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5b418c17084420b867498fda79fb0a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:24.391183 919 raft_consensus.cc:2463] T 3f5b418c17084420b867498fda79fb0a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:24.391572 525 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet bf58fb4538d946e1818a5d31d2a988e2
I20250112 02:52:24.391650 713 leader_election.cc:304] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
W20250112 02:52:24.392151 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_672d7452ad6d4c99ae9f2ae77b592f78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.393110 1193 raft_consensus.cc:2798] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:24.393260 1193 raft_consensus.cc:491] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:24.393352 1193 raft_consensus.cc:3054] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:24.394459 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b521522c4784d2ab4056268823f1099 with delete_type TABLET_DATA_DELETED (replaced by tablet ab2e31a65294407bb62b13e92898a7d2) from {username='slave'} at 127.0.0.1:50844
W20250112 02:52:24.395792 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c69587c5a09d40f0baba5c2f6e8a7b41 consensus state has no leader
I20250112 02:52:24.396652 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_426c5d1d19a94f0d8110a9cbee9a9044"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.397094 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_426c5d1d19a94f0d8110a9cbee9a9044 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.397518 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet d3e41e307df64109a41b0c1c44d3bda6
I20250112 02:52:24.400640 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_84c29ed08966455b9dde3dfcc832ffac" } modify_external_catalogs: true
I20250112 02:52:24.400876 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_84c29ed08966455b9dde3dfcc832ffac" } modify_external_catalogs: true
I20250112 02:52:24.401973 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_edaa8b50dfa8463e99e05ab8d1d89bc3" } modify_external_catalogs: true
I20250112 02:52:24.402155 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_edaa8b50dfa8463e99e05ab8d1d89bc3" } modify_external_catalogs: true
I20250112 02:52:24.403756 896 tablet_service.cc:1514] Processing DeleteTablet for tablet d3e41e307df64109a41b0c1c44d3bda6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:24.405984 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 71b5f80ed198491780cbe17e6cd3fa15 consensus state has no leader
I20250112 02:52:24.406286 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 2728a022ef0d4952966d397a32356503 with delete_type TABLET_DATA_DELETED (replaced by tablet bf58fb4538d946e1818a5d31d2a988e2) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.407016 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5b418c17084420b867498fda79fb0a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:24.407498 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet d3e41e307df64109a41b0c1c44d3bda6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.408280 1175 ts_tablet_manager.cc:1918] T 6834be02864543469056a928fafe28ca P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.408298 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 2728a022ef0d4952966d397a32356503 with delete_type TABLET_DATA_DELETED (replaced by tablet bf58fb4538d946e1818a5d31d2a988e2) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.410290 1175 log.cc:1198] T 6834be02864543469056a928fafe28ca P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/6834be02864543469056a928fafe28ca
I20250112 02:52:24.410715 1175 ts_tablet_manager.cc:1939] T 6834be02864543469056a928fafe28ca P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:24.412518 524 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 63a432747dcf41bcace55c8fb80c7265
I20250112 02:52:24.412850 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet b885247d2cb449e0a29f6f5d81518589
I20250112 02:52:24.413934 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 6834be02864543469056a928fafe28ca (table default.Table_5fe703bbcb87402b87a82e1441569d53 [id=a2449caa011d43bfb54aea196629c9f1]) successfully deleted
I20250112 02:52:24.414134 899 tablet_service.cc:1514] Processing DeleteTablet for tablet b885247d2cb449e0a29f6f5d81518589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.410183 1181 tablet_bootstrap.cc:492] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:24.414552 1181 ts_tablet_manager.cc:1397] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.191s user 0.004s sys 0.015s
I20250112 02:52:24.414727 1173 tablet_bootstrap.cc:492] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:24.414886 1173 ts_tablet_manager.cc:1397] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.079s user 0.000s sys 0.002s
I20250112 02:52:24.415771 1173 raft_consensus.cc:357] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:24.415973 1173 raft_consensus.cc:383] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.416065 1173 raft_consensus.cc:738] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.416303 1173 consensus_queue.cc:260] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:24.416107 1181 raft_consensus.cc:357] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:24.417219 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet aba1744138fe42ae9d097db01cac083d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aba1744138fe42ae9d097db01cac083d already in progress: creating tablet
I20250112 02:52:24.417496 1181 raft_consensus.cc:383] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.417804 1181 raft_consensus.cc:738] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
W20250112 02:52:24.418421 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 6b521522c4784d2ab4056268823f1099 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b521522c4784d2ab4056268823f1099 already in progress: creating tablet
I20250112 02:52:24.419859 1171 ts_tablet_manager.cc:1918] T 2b34fb96bc5140118725af8346ea827f P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.420011 1171 log.cc:1198] T 2b34fb96bc5140118725af8346ea827f P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/2b34fb96bc5140118725af8346ea827f
I20250112 02:52:24.420254 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bca7a6cecdc49aeb9aa4ae65beb7757 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:22 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.420423 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet b885247d2cb449e0a29f6f5d81518589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.420825 1173 ts_tablet_manager.cc:1428] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250112 02:52:24.418923 1181 consensus_queue.cc:260] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:24.421515 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet d3e41e307df64109a41b0c1c44d3bda6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3e41e307df64109a41b0c1c44d3bda6 already in progress: creating tablet
W20250112 02:52:24.421819 714 leader_election.cc:343] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:24.421829 760 tablet_service.cc:1514] Processing DeleteTablet for tablet d3e41e307df64109a41b0c1c44d3bda6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.423118 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b521522c4784d2ab4056268823f1099 with delete_type TABLET_DATA_DELETED (replaced by tablet ab2e31a65294407bb62b13e92898a7d2) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.423840 1193 raft_consensus.cc:513] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.424459 1193 leader_election.cc:290] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:24.425108 1173 tablet_bootstrap.cc:492] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:24.426235 754 tablet_service.cc:1514] Processing DeleteTablet for tablet b885247d2cb449e0a29f6f5d81518589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.426333 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5b418c17084420b867498fda79fb0a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:24.426795 714 leader_election.cc:343] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:24.427338 1171 ts_tablet_manager.cc:1939] T 2b34fb96bc5140118725af8346ea827f P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:24.428499 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 2b34fb96bc5140118725af8346ea827f (table default.Table_c2d18d092b5a46f8b59f64aa4a8f02c7 [id=059958c4e1c74d8d8e4c029030d85895]) successfully deleted
W20250112 02:52:24.429265 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d3e41e307df64109a41b0c1c44d3bda6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3e41e307df64109a41b0c1c44d3bda6 already in progress: creating tablet
W20250112 02:52:24.429493 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 6b521522c4784d2ab4056268823f1099 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b521522c4784d2ab4056268823f1099 already in progress: creating tablet
W20250112 02:52:24.429778 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b885247d2cb449e0a29f6f5d81518589 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b885247d2cb449e0a29f6f5d81518589 already in progress: creating tablet
I20250112 02:52:24.430532 1177 tablet_bootstrap.cc:492] T 22750265b522471680835e7f98475e6e P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:24.430663 1177 ts_tablet_manager.cc:1397] T 22750265b522471680835e7f98475e6e P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.098s user 0.002s sys 0.007s
I20250112 02:52:24.431350 1177 raft_consensus.cc:357] T 22750265b522471680835e7f98475e6e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.431550 1177 raft_consensus.cc:383] T 22750265b522471680835e7f98475e6e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.431636 1177 raft_consensus.cc:738] T 22750265b522471680835e7f98475e6e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.431888 1177 consensus_queue.cc:260] T 22750265b522471680835e7f98475e6e P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.432188 754 tablet_service.cc:1467] Processing CreateTablet for tablet 9163878cfeb64693a9c5493e812546af (DEFAULT_TABLE table=default.Table_426c5d1d19a94f0d8110a9cbee9a9044 [id=69f55e532f714a2d84790df8ff19e6dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.434450 1177 ts_tablet_manager.cc:1428] T 22750265b522471680835e7f98475e6e P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:24.434690 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9163878cfeb64693a9c5493e812546af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.434854 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 22750265b522471680835e7f98475e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.435360 1171 tablet_replica.cc:331] T 22750265b522471680835e7f98475e6e P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:24.435292 750 tablet_service.cc:1467] Processing CreateTablet for tablet fb0aba0139cf479b80b32f3dfc936535 (DEFAULT_TABLE table=default.Table_672d7452ad6d4c99ae9f2ae77b592f78 [id=33dad7ab4106404aaf33266ed1ea5635]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.435642 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb0aba0139cf479b80b32f3dfc936535. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.437348 760 tablet_service.cc:1467] Processing CreateTablet for tablet 8a88463991294bd8a335337f8b072c3d (DEFAULT_TABLE table=default.Table_56ea07abda424090a7942efabc871092 [id=44f7673679d14145a8cf65abd82094ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.437712 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a88463991294bd8a335337f8b072c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.437904 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 9163878cfeb64693a9c5493e812546af (DEFAULT_TABLE table=default.Table_426c5d1d19a94f0d8110a9cbee9a9044 [id=69f55e532f714a2d84790df8ff19e6dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.438347 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9163878cfeb64693a9c5493e812546af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.440050 1034 tablet_service.cc:1467] Processing CreateTablet for tablet fb0aba0139cf479b80b32f3dfc936535 (DEFAULT_TABLE table=default.Table_672d7452ad6d4c99ae9f2ae77b592f78 [id=33dad7ab4106404aaf33266ed1ea5635]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.440519 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb0aba0139cf479b80b32f3dfc936535. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.442485 1171 raft_consensus.cc:2238] T 22750265b522471680835e7f98475e6e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.442687 1171 raft_consensus.cc:2267] T 22750265b522471680835e7f98475e6e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.443329 1171 ts_tablet_manager.cc:1905] T 22750265b522471680835e7f98475e6e P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.445770 760 tablet_service.cc:1467] Processing CreateTablet for tablet ab2e31a65294407bb62b13e92898a7d2 (DEFAULT_TABLE table=default.Table_a60b0bb4ae784e09bb62235bfb1d7bcd [id=f8cd4289f89a4a748c26a7a01236406e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.446179 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab2e31a65294407bb62b13e92898a7d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.425043 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5b418c17084420b867498fda79fb0a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:24.446616 919 raft_consensus.cc:3054] T 3f5b418c17084420b867498fda79fb0a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:24.433866 899 tablet_service.cc:1467] Processing CreateTablet for tablet 9163878cfeb64693a9c5493e812546af (DEFAULT_TABLE table=default.Table_426c5d1d19a94f0d8110a9cbee9a9044 [id=69f55e532f714a2d84790df8ff19e6dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.447533 1172 ts_tablet_manager.cc:1918] T d9125dd3f9544a54ac373ae0f90cce5f P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.447661 1172 log.cc:1198] T d9125dd3f9544a54ac373ae0f90cce5f P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/d9125dd3f9544a54ac373ae0f90cce5f
I20250112 02:52:24.447772 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9163878cfeb64693a9c5493e812546af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.447962 1172 ts_tablet_manager.cc:1939] T d9125dd3f9544a54ac373ae0f90cce5f P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:24.448838 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet d9125dd3f9544a54ac373ae0f90cce5f (table default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94]) successfully deleted
I20250112 02:52:24.450075 1173 tablet_bootstrap.cc:654] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.452066 522 master_service.cc:946] ReplaceTablet: received request to replace tablet 71b5f80ed198491780cbe17e6cd3fa15 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:24.452564 897 tablet_service.cc:1467] Processing CreateTablet for tablet ab2e31a65294407bb62b13e92898a7d2 (DEFAULT_TABLE table=default.Table_a60b0bb4ae784e09bb62235bfb1d7bcd [id=f8cd4289f89a4a748c26a7a01236406e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.452903 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab2e31a65294407bb62b13e92898a7d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.455196 919 raft_consensus.cc:2463] T 3f5b418c17084420b867498fda79fb0a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:24.455587 713 leader_election.cc:304] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:24.455996 1193 raft_consensus.cc:2798] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:24.442270 896 tablet_service.cc:1467] Processing CreateTablet for tablet fb0aba0139cf479b80b32f3dfc936535 (DEFAULT_TABLE table=default.Table_672d7452ad6d4c99ae9f2ae77b592f78 [id=33dad7ab4106404aaf33266ed1ea5635]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.444952 893 tablet_service.cc:1467] Processing CreateTablet for tablet 8a88463991294bd8a335337f8b072c3d (DEFAULT_TABLE table=default.Table_56ea07abda424090a7942efabc871092 [id=44f7673679d14145a8cf65abd82094ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.457408 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a88463991294bd8a335337f8b072c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.460932 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 71b5f80ed198491780cbe17e6cd3fa15
I20250112 02:52:24.461347 522 catalog_manager.cc:6596] ReplaceTablet: tablet 71b5f80ed198491780cbe17e6cd3fa15 deleted and replaced by tablet c7c530aa0aa64e47ae27d61f341d2bd9
I20250112 02:52:24.461555 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 71b5f80ed198491780cbe17e6cd3fa15 with delete_type TABLET_DATA_DELETED (replaced by tablet c7c530aa0aa64e47ae27d61f341d2bd9) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.462045 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 71b5f80ed198491780cbe17e6cd3fa15 with delete_type TABLET_DATA_DELETED (replaced by tablet c7c530aa0aa64e47ae27d61f341d2bd9) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.464147 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb0aba0139cf479b80b32f3dfc936535. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.466123 754 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0c2fad4f3243238641aae56b305bb5 (DEFAULT_TABLE table=default.Table_82662e02aee0459dad1b621df36f58f0 [id=50a3f0607e95430594c6769983f4e31b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.466385 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 71b5f80ed198491780cbe17e6cd3fa15 with delete_type TABLET_DATA_DELETED (replaced by tablet c7c530aa0aa64e47ae27d61f341d2bd9) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.466452 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c0c2fad4f3243238641aae56b305bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.466671 897 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0c2fad4f3243238641aae56b305bb5 (DEFAULT_TABLE table=default.Table_82662e02aee0459dad1b621df36f58f0 [id=50a3f0607e95430594c6769983f4e31b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.466992 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c0c2fad4f3243238641aae56b305bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.468034 522 master_service.cc:946] ReplaceTablet: received request to replace tablet 49650a419dfd49d5a1547770d3f2fe54 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:24.468181 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0c2fad4f3243238641aae56b305bb5 (DEFAULT_TABLE table=default.Table_82662e02aee0459dad1b621df36f58f0 [id=50a3f0607e95430594c6769983f4e31b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.468459 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c0c2fad4f3243238641aae56b305bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.446460 1175 tablet_replica.cc:331] T 2728a022ef0d4952966d397a32356503 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:24.473282 524 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 (table default.Table_e8041534ae864796a75a6edf2e4a33dc [id=5c4769bfcefd4fe0a0f3c2e315d50c32]). Expected version 3 got 0
I20250112 02:52:24.473711 1020 tablet_service.cc:1467] Processing CreateTablet for tablet ab2e31a65294407bb62b13e92898a7d2 (DEFAULT_TABLE table=default.Table_a60b0bb4ae784e09bb62235bfb1d7bcd [id=f8cd4289f89a4a748c26a7a01236406e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.474042 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab2e31a65294407bb62b13e92898a7d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.475301 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 49650a419dfd49d5a1547770d3f2fe54
I20250112 02:52:24.475569 1181 ts_tablet_manager.cc:1428] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.061s user 0.002s sys 0.000s
I20250112 02:52:24.475818 522 catalog_manager.cc:6596] ReplaceTablet: tablet 49650a419dfd49d5a1547770d3f2fe54 deleted and replaced by tablet d8eb0efc5d7f4c72bdfc7531a444d42a
I20250112 02:52:24.475836 1175 raft_consensus.cc:2238] T 2728a022ef0d4952966d397a32356503 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.476488 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 49650a419dfd49d5a1547770d3f2fe54 with delete_type TABLET_DATA_DELETED (replaced by tablet d8eb0efc5d7f4c72bdfc7531a444d42a) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.476501 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 49650a419dfd49d5a1547770d3f2fe54 with delete_type TABLET_DATA_DELETED (replaced by tablet d8eb0efc5d7f4c72bdfc7531a444d42a) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.476755 1175 raft_consensus.cc:2267] T 2728a022ef0d4952966d397a32356503 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.476958 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 49650a419dfd49d5a1547770d3f2fe54 with delete_type TABLET_DATA_DELETED (replaced by tablet d8eb0efc5d7f4c72bdfc7531a444d42a) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.477653 1181 tablet_bootstrap.cc:492] T e8220011788a480eb50ee4b670c45739 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:24.479422 1175 ts_tablet_manager.cc:1905] T 2728a022ef0d4952966d397a32356503 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.479593 522 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 (table default.Table_e8041534ae864796a75a6edf2e4a33dc [id=5c4769bfcefd4fe0a0f3c2e315d50c32]). Expected version 3 got 0
I20250112 02:52:24.479784 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 8a88463991294bd8a335337f8b072c3d (DEFAULT_TABLE table=default.Table_56ea07abda424090a7942efabc871092 [id=44f7673679d14145a8cf65abd82094ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.480708 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a88463991294bd8a335337f8b072c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.481443 754 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd498ccce484a89855c982762086890 (DEFAULT_TABLE table=default.Table_945067c243f84394957a328d65592441 [id=5a5f3290dab54cf0b13c88efd52c90c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.481851 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd498ccce484a89855c982762086890. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.482033 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_12a25f7a620047b69f34be9eeae64dd2" } modify_external_catalogs: true
I20250112 02:52:24.482216 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_12a25f7a620047b69f34be9eeae64dd2" } modify_external_catalogs: true
I20250112 02:52:24.483952 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd498ccce484a89855c982762086890 (DEFAULT_TABLE table=default.Table_945067c243f84394957a328d65592441 [id=5a5f3290dab54cf0b13c88efd52c90c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.484301 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd498ccce484a89855c982762086890. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.484540 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_cfef2f0eb8dd4927b6d0af0a8c3c739c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.485047 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cfef2f0eb8dd4927b6d0af0a8c3c739c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.486811 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet b2d8ea2ef18a46508371b4789ab6f4fa
I20250112 02:52:24.487547 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet b2d8ea2ef18a46508371b4789ab6f4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.488324 1181 tablet_bootstrap.cc:654] T e8220011788a480eb50ee4b670c45739 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.491925 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_916fd232386d4e659e4392822be45c8f" } new_table_name: "default.Table_b56246b266524f2eb968fe7df99ce8b2" modify_external_catalogs: true
I20250112 02:52:24.492125 899 tablet_service.cc:1514] Processing DeleteTablet for tablet b2d8ea2ef18a46508371b4789ab6f4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:24.492214 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 328e307a4b35445f92cbfab24f2ada48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 328e307a4b35445f92cbfab24f2ada48 already in progress: creating tablet
I20250112 02:52:24.493460 1031 tablet_service.cc:1467] Processing CreateTablet for tablet b7bd96fb381f4e73a0081be3fc799c36 (DEFAULT_TABLE table=default.Table_893bf955f62b495c9dcc59390b472f96 [id=e992deac51c64dd8a022428dd3c53714]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.493885 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7bd96fb381f4e73a0081be3fc799c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.494086 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 27ab58e45d0d44b283019240f44b66bf (DEFAULT_TABLE table=default.Table_d588e6b08cae48e590a4a57002df6323 [id=7a7301e1e2d74a6d9340ee996656f264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.494791 753 tablet_service.cc:1467] Processing CreateTablet for tablet b7bd96fb381f4e73a0081be3fc799c36 (DEFAULT_TABLE table=default.Table_893bf955f62b495c9dcc59390b472f96 [id=e992deac51c64dd8a022428dd3c53714]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.494871 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27ab58e45d0d44b283019240f44b66bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.495612 897 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd498ccce484a89855c982762086890 (DEFAULT_TABLE table=default.Table_945067c243f84394957a328d65592441 [id=5a5f3290dab54cf0b13c88efd52c90c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.495982 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_21f86d73f86a49d5aeb339558a9f2b67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:24.496089 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_57a7c003d55441cc9b836feecff7ca2f" } new_table_name: "default.Table_4e7e1a3381d2412987a7061d9030f9f9" modify_external_catalogs: true
W20250112 02:52:24.496474 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21f86d73f86a49d5aeb339558a9f2b67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.497601 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd498ccce484a89855c982762086890. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.499109 899 tablet_service.cc:1467] Processing CreateTablet for tablet b7bd96fb381f4e73a0081be3fc799c36 (DEFAULT_TABLE table=default.Table_893bf955f62b495c9dcc59390b472f96 [id=e992deac51c64dd8a022428dd3c53714]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.499938 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7bd96fb381f4e73a0081be3fc799c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.499961 896 tablet_service.cc:1467] Processing CreateTablet for tablet 27ab58e45d0d44b283019240f44b66bf (DEFAULT_TABLE table=default.Table_d588e6b08cae48e590a4a57002df6323 [id=7a7301e1e2d74a6d9340ee996656f264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.500492 759 tablet_service.cc:1467] Processing CreateTablet for tablet 27ab58e45d0d44b283019240f44b66bf (DEFAULT_TABLE table=default.Table_d588e6b08cae48e590a4a57002df6323 [id=7a7301e1e2d74a6d9340ee996656f264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.500770 750 tablet_service.cc:1514] Processing DeleteTablet for tablet b2d8ea2ef18a46508371b4789ab6f4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:24.502254 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99715f37fbef47c6860fa1d95e350f20 consensus state has no leader
I20250112 02:52:24.501217 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27ab58e45d0d44b283019240f44b66bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.502961 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7bd96fb381f4e73a0081be3fc799c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.503172 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27ab58e45d0d44b283019240f44b66bf. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:24.504479 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 97b31ebe035c4552baaffc3133ec73c1 consensus state has no leader
I20250112 02:52:24.512486 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_3530ac60f1e04607aafce2de389968df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.512997 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3530ac60f1e04607aafce2de389968df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.526496 1172 tablet_replica.cc:331] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:24.526693 1172 raft_consensus.cc:2238] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.526868 1172 raft_consensus.cc:2267] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.527441 1193 raft_consensus.cc:695] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:24.527635 1172 ts_tablet_manager.cc:1905] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.527995 1193 consensus_queue.cc:237] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.531292 759 tablet_service.cc:1467] Processing CreateTablet for tablet 35f59617b34046568dd83bbb7cdc42d2 (DEFAULT_TABLE table=default.Table_cfef2f0eb8dd4927b6d0af0a8c3c739c [id=a0145eb8ddc14728b448e5c08a4d9311]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.531683 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35f59617b34046568dd83bbb7cdc42d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.532509 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 35f59617b34046568dd83bbb7cdc42d2 (DEFAULT_TABLE table=default.Table_cfef2f0eb8dd4927b6d0af0a8c3c739c [id=a0145eb8ddc14728b448e5c08a4d9311]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.532670 899 tablet_service.cc:1467] Processing CreateTablet for tablet 35f59617b34046568dd83bbb7cdc42d2 (DEFAULT_TABLE table=default.Table_cfef2f0eb8dd4927b6d0af0a8c3c739c [id=a0145eb8ddc14728b448e5c08a4d9311]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.532999 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35f59617b34046568dd83bbb7cdc42d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.532924 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35f59617b34046568dd83bbb7cdc42d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.536787 1173 tablet_bootstrap.cc:492] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:24.536952 1173 ts_tablet_manager.cc:1397] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.112s user 0.000s sys 0.003s
I20250112 02:52:24.537159 893 tablet_service.cc:1467] Processing CreateTablet for tablet c7c530aa0aa64e47ae27d61f341d2bd9 (DEFAULT_TABLE table=default.Table_247a821819994de6adac6fabd06c3a1f [id=60486b9e81254d74b8e08a1cadde13bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.537456 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c530aa0aa64e47ae27d61f341d2bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.537765 1173 raft_consensus.cc:357] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.537968 1173 raft_consensus.cc:383] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.538028 1173 raft_consensus.cc:738] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.537712 526 catalog_manager.cc:5526] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:24.538264 1173 consensus_queue.cc:260] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.538564 526 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3f5b418c17084420b867498fda79fb0a (table default.Table_0b78c0ecb1f245838a0117009873c0e9 [id=ad0be37e6b6f4b9597dcb4cbd50d5b12]). Expected version 1 got 0
I20250112 02:52:24.539958 753 tablet_service.cc:1467] Processing CreateTablet for tablet c7c530aa0aa64e47ae27d61f341d2bd9 (DEFAULT_TABLE table=default.Table_247a821819994de6adac6fabd06c3a1f [id=60486b9e81254d74b8e08a1cadde13bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.540380 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c530aa0aa64e47ae27d61f341d2bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.541385 1173 ts_tablet_manager.cc:1428] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 02:52:24.541915 760 tablet_service.cc:1467] Processing CreateTablet for tablet d8eb0efc5d7f4c72bdfc7531a444d42a (DEFAULT_TABLE table=default.Table_d561e1740cc545dc8aa4babb76b45c37 [id=de27aeff6b3a49b3956498ea5015ff33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.542970 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8eb0efc5d7f4c72bdfc7531a444d42a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.543385 1177 tablet_bootstrap.cc:492] T bdaec169cfb5471391fcd3050bb00431 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:24.545423 1177 tablet_bootstrap.cc:654] T bdaec169cfb5471391fcd3050bb00431 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.548043 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_44769a7c8b7e463dbe5d907dacc0410f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:24.547904 1171 ts_tablet_manager.cc:1918] T 22750265b522471680835e7f98475e6e P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.548353 1171 log.cc:1198] T 22750265b522471680835e7f98475e6e P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/22750265b522471680835e7f98475e6e
W20250112 02:52:24.548561 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44769a7c8b7e463dbe5d907dacc0410f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.548651 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_21f86d73f86a49d5aeb339558a9f2b67" } new_table_name: "default.Table_a6b320592c334f8bb0ff1a1ddf3ab0c0" modify_external_catalogs: true
I20250112 02:52:24.554437 1181 tablet_bootstrap.cc:492] T e8220011788a480eb50ee4b670c45739 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:24.554773 1181 ts_tablet_manager.cc:1397] T e8220011788a480eb50ee4b670c45739 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.077s user 0.003s sys 0.000s
I20250112 02:52:24.555272 1171 ts_tablet_manager.cc:1939] T 22750265b522471680835e7f98475e6e P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:24.556260 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 22750265b522471680835e7f98475e6e (table default.Table_89761f48fc8547e8b786c9559485f243 [id=c8799d47318d42d9859e36a37a5a9445]) successfully deleted
W20250112 02:52:24.556941 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 71b5f80ed198491780cbe17e6cd3fa15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71b5f80ed198491780cbe17e6cd3fa15 already in progress: creating tablet
I20250112 02:52:24.557711 1181 raft_consensus.cc:357] T e8220011788a480eb50ee4b670c45739 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.558013 1181 raft_consensus.cc:383] T e8220011788a480eb50ee4b670c45739 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.558094 1181 raft_consensus.cc:738] T e8220011788a480eb50ee4b670c45739 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.558396 1181 consensus_queue.cc:260] T e8220011788a480eb50ee4b670c45739 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.560070 522 master_service.cc:946] ReplaceTablet: received request to replace tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 from {username='slave'} at 127.0.0.1:40934
W20250112 02:52:24.560362 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 65f22a7b1b804661af3b7e785a833739 has no consensus state
I20250112 02:52:24.554205 1173 tablet_bootstrap.cc:492] T 3f5b418c17084420b867498fda79fb0a P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:24.561321 1178 raft_consensus.cc:491] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:24.561506 1178 raft_consensus.cc:513] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.562110 1178 leader_election.cc:290] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:24.562264 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 49650a419dfd49d5a1547770d3f2fe54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49650a419dfd49d5a1547770d3f2fe54 already in progress: creating tablet
W20250112 02:52:24.562973 989 leader_election.cc:343] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: e56a00ec3f5040b982c555d65f07d34d
I20250112 02:52:24.563133 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_6255caf5a65d4d9192e68f39413398f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.563345 991 leader_election.cc:343] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: e56a00ec3f5040b982c555d65f07d34d
I20250112 02:52:24.563457 991 leader_election.cc:304] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
W20250112 02:52:24.563575 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6255caf5a65d4d9192e68f39413398f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.564097 1178 raft_consensus.cc:2743] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:24.564815 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b2d8ea2ef18a46508371b4789ab6f4fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2d8ea2ef18a46508371b4789ab6f4fa already in progress: creating tablet
I20250112 02:52:24.565547 1173 tablet_bootstrap.cc:654] T 3f5b418c17084420b867498fda79fb0a P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.565583 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 325e43fdf2fc4eacb48a5227e2ff3bf7
I20250112 02:52:24.566351 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c686fdc8896463c81c9f7c5f53a3da2) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.566642 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c686fdc8896463c81c9f7c5f53a3da2) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.566880 522 catalog_manager.cc:6596] ReplaceTablet: tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 deleted and replaced by tablet 3c686fdc8896463c81c9f7c5f53a3da2
I20250112 02:52:24.566919 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c686fdc8896463c81c9f7c5f53a3da2) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.568532 1181 ts_tablet_manager.cc:1428] T e8220011788a480eb50ee4b670c45739 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250112 02:52:24.568917 1181 tablet_bootstrap.cc:492] T 235e9dd0964c4ea6af7d05ef58acae0d P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:24.569911 1020 tablet_service.cc:1467] Processing CreateTablet for tablet c7c530aa0aa64e47ae27d61f341d2bd9 (DEFAULT_TABLE table=default.Table_247a821819994de6adac6fabd06c3a1f [id=60486b9e81254d74b8e08a1cadde13bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.571399 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet e56a00ec3f5040b982c555d65f07d34d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:22 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.570451 897 tablet_service.cc:1467] Processing CreateTablet for tablet d8eb0efc5d7f4c72bdfc7531a444d42a (DEFAULT_TABLE table=default.Table_d561e1740cc545dc8aa4babb76b45c37 [id=de27aeff6b3a49b3956498ea5015ff33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.571833 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8eb0efc5d7f4c72bdfc7531a444d42a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.572232 1181 tablet_bootstrap.cc:654] T 235e9dd0964c4ea6af7d05ef58acae0d P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.573864 1177 tablet_bootstrap.cc:492] T bdaec169cfb5471391fcd3050bb00431 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:24.574064 1177 ts_tablet_manager.cc:1397] T bdaec169cfb5471391fcd3050bb00431 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20250112 02:52:24.574359 1029 tablet_service.cc:1467] Processing CreateTablet for tablet d8eb0efc5d7f4c72bdfc7531a444d42a (DEFAULT_TABLE table=default.Table_d561e1740cc545dc8aa4babb76b45c37 [id=de27aeff6b3a49b3956498ea5015ff33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.574880 1177 raft_consensus.cc:357] T bdaec169cfb5471391fcd3050bb00431 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.575125 1177 raft_consensus.cc:383] T bdaec169cfb5471391fcd3050bb00431 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.575223 1177 raft_consensus.cc:738] T bdaec169cfb5471391fcd3050bb00431 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.575531 1177 consensus_queue.cc:260] T bdaec169cfb5471391fcd3050bb00431 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.576931 1177 ts_tablet_manager.cc:1428] T bdaec169cfb5471391fcd3050bb00431 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 02:52:24.577412 1177 tablet_bootstrap.cc:492] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:24.579380 893 tablet_service.cc:1514] Processing DeleteTablet for tablet e8220011788a480eb50ee4b670c45739 with delete_type TABLET_DATA_DELETED (replaced by tablet b885247d2cb449e0a29f6f5d81518589) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.579843 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_11b4065cfb0449b68950d63e65f566a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.580302 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11b4065cfb0449b68950d63e65f566a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:24.582041 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 already in progress: creating tablet
I20250112 02:52:24.583590 754 tablet_service.cc:1514] Processing DeleteTablet for tablet bdaec169cfb5471391fcd3050bb00431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.584584 1171 tablet_replica.cc:331] T bdaec169cfb5471391fcd3050bb00431 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:24.584756 1171 raft_consensus.cc:2238] T bdaec169cfb5471391fcd3050bb00431 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.584913 1171 raft_consensus.cc:2267] T bdaec169cfb5471391fcd3050bb00431 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.585592 1171 ts_tablet_manager.cc:1905] T bdaec169cfb5471391fcd3050bb00431 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.580732 1175 ts_tablet_manager.cc:1918] T 2728a022ef0d4952966d397a32356503 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.587762 1175 log.cc:1198] T 2728a022ef0d4952966d397a32356503 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/2728a022ef0d4952966d397a32356503
I20250112 02:52:24.588100 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c530aa0aa64e47ae27d61f341d2bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.588109 1175 ts_tablet_manager.cc:1939] T 2728a022ef0d4952966d397a32356503 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:24.588316 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8eb0efc5d7f4c72bdfc7531a444d42a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:24.591948 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet b885247d2cb449e0a29f6f5d81518589 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b885247d2cb449e0a29f6f5d81518589 already in progress: creating tablet
W20250112 02:52:24.592586 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 71b5f80ed198491780cbe17e6cd3fa15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71b5f80ed198491780cbe17e6cd3fa15 already in progress: creating tablet
I20250112 02:52:24.592959 1177 tablet_bootstrap.cc:654] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:24.596105 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 49650a419dfd49d5a1547770d3f2fe54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49650a419dfd49d5a1547770d3f2fe54 already in progress: creating tablet
W20250112 02:52:24.596659 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet b2d8ea2ef18a46508371b4789ab6f4fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2d8ea2ef18a46508371b4789ab6f4fa already in progress: creating tablet
I20250112 02:52:24.597503 1175 tablet_replica.cc:331] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:24.597719 1175 raft_consensus.cc:2238] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.597932 1175 raft_consensus.cc:2267] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.598598 1175 ts_tablet_manager.cc:1905] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.603754 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_e8041534ae864796a75a6edf2e4a33dc" } new_table_name: "default.Table_f7f876c63e68490c805c0bfa3f0c25ea" modify_external_catalogs: true
I20250112 02:52:24.603830 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_bd1cdf8c9b3343af8d17a9fd05c77145" } modify_external_catalogs: true
I20250112 02:52:24.604017 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_952149cc55fa44fdbae173c5e83bc4f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.604467 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_952149cc55fa44fdbae173c5e83bc4f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.605592 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_fd0169db0f984d3cbc0d8624dda686bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.606079 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd0169db0f984d3cbc0d8624dda686bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.607105 1177 tablet_bootstrap.cc:492] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:24.607216 1177 ts_tablet_manager.cc:1397] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.001s
I20250112 02:52:24.607209 520 master_service.cc:946] ReplaceTablet: received request to replace tablet f9186ac46ed441ee9308490e90da0e23 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:24.607761 521 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_6255caf5a65d4d9192e68f39413398f0" } modify_external_catalogs: true
I20250112 02:52:24.607884 1177 raft_consensus.cc:357] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.608457 1177 raft_consensus.cc:383] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.608445 521 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_6255caf5a65d4d9192e68f39413398f0" } modify_external_catalogs: true
I20250112 02:52:24.608561 1177 raft_consensus.cc:738] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.608752 1177 consensus_queue.cc:260] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.604023 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_bd1cdf8c9b3343af8d17a9fd05c77145" } modify_external_catalogs: true
I20250112 02:52:24.609516 1177 ts_tablet_manager.cc:1428] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:24.609794 1177 tablet_bootstrap.cc:492] T 6834be02864543469056a928fafe28ca P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:24.610145 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_32538719e60a4843b560a8a6f3ac17b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:24.610248 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_9ff5918b8b90482c9b89b81bdf7183e6" } new_table_name: "default.Table_4c59f5d61aab4b6184f0d9a4e21f965f" modify_external_catalogs: true
W20250112 02:52:24.610601 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32538719e60a4843b560a8a6f3ac17b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.611843 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_f3695b00a89b4353903587a6c1418d8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.612247 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3695b00a89b4353903587a6c1418d8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:24.614005 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c686fdc8896463c81c9f7c5f53a3da2 has no consensus state
I20250112 02:52:24.614063 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_b56246b266524f2eb968fe7df99ce8b2" } modify_external_catalogs: true
I20250112 02:52:24.614238 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_b56246b266524f2eb968fe7df99ce8b2" } modify_external_catalogs: true
I20250112 02:52:24.616146 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_3530ac60f1e04607aafce2de389968df" } modify_external_catalogs: true
I20250112 02:52:24.616312 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_3530ac60f1e04607aafce2de389968df" } modify_external_catalogs: true
I20250112 02:52:24.618047 919 raft_consensus.cc:1270] T 3f5b418c17084420b867498fda79fb0a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Refusing update from remote peer 289afc116aa74408aa5bff74e11c50d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 02:52:24.618958 1193 consensus_queue.cc:1035] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:24.620163 897 tablet_service.cc:1467] Processing CreateTablet for tablet b9199b0d2b924a79938a979df4196bb4 (DEFAULT_TABLE table=default.Table_44769a7c8b7e463dbe5d907dacc0410f [id=472b3381fca44299922af20255b6aad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.621460 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9199b0d2b924a79938a979df4196bb4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:24.622283 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 96e201d798c94e448a78e1c8adf41724 consensus state has no leader
I20250112 02:52:24.622546 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c686fdc8896463c81c9f7c5f53a3da2) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.623054 760 tablet_service.cc:1467] Processing CreateTablet for tablet 65f22a7b1b804661af3b7e785a833739 (DEFAULT_TABLE table=default.Table_a6b320592c334f8bb0ff1a1ddf3ab0c0 [id=87b950688174482d8509a759d3c6e28f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.624742 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f22a7b1b804661af3b7e785a833739. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.627291 1171 ts_tablet_manager.cc:1918] T bdaec169cfb5471391fcd3050bb00431 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.629454 1181 tablet_bootstrap.cc:492] T 235e9dd0964c4ea6af7d05ef58acae0d P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:24.629570 1181 ts_tablet_manager.cc:1397] T 235e9dd0964c4ea6af7d05ef58acae0d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.000s
I20250112 02:52:24.630239 1181 raft_consensus.cc:357] T 235e9dd0964c4ea6af7d05ef58acae0d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.630524 1181 raft_consensus.cc:383] T 235e9dd0964c4ea6af7d05ef58acae0d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.630599 1181 raft_consensus.cc:738] T 235e9dd0964c4ea6af7d05ef58acae0d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.630905 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 65f22a7b1b804661af3b7e785a833739 (DEFAULT_TABLE table=default.Table_a6b320592c334f8bb0ff1a1ddf3ab0c0 [id=87b950688174482d8509a759d3c6e28f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.630932 1181 consensus_queue.cc:260] T 235e9dd0964c4ea6af7d05ef58acae0d P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.631338 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f22a7b1b804661af3b7e785a833739. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.631634 1181 ts_tablet_manager.cc:1428] T 235e9dd0964c4ea6af7d05ef58acae0d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:24.631906 1181 tablet_bootstrap.cc:492] T aa67bf25dc894d7ea96d247a22c14e03 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:24.633563 1181 tablet_bootstrap.cc:654] T aa67bf25dc894d7ea96d247a22c14e03 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:24.641214 714 consensus_peers.cc:487] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8 -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:24.641400 899 tablet_service.cc:1467] Processing CreateTablet for tablet 65f22a7b1b804661af3b7e785a833739 (DEFAULT_TABLE table=default.Table_a6b320592c334f8bb0ff1a1ddf3ab0c0 [id=87b950688174482d8509a759d3c6e28f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.641714 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f22a7b1b804661af3b7e785a833739. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.646368 526 master_service.cc:946] ReplaceTablet: received request to replace tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:24.646672 1171 log.cc:1198] T bdaec169cfb5471391fcd3050bb00431 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/bdaec169cfb5471391fcd3050bb00431
I20250112 02:52:24.647017 1171 ts_tablet_manager.cc:1939] T bdaec169cfb5471391fcd3050bb00431 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:24.645852 1172 ts_tablet_manager.cc:1918] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.647207 1172 log.cc:1198] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/9c9a803ff19645328107d3b2fabb90c6
I20250112 02:52:24.647595 1172 ts_tablet_manager.cc:1939] T 9c9a803ff19645328107d3b2fabb90c6 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:24.647969 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet bdaec169cfb5471391fcd3050bb00431 (table default.Table_7d31752a7e7246ad9806310039f73c14 [id=62176e03ce0b4ced96e5083fa32b1826]) successfully deleted
I20250112 02:52:24.624384 1029 tablet_service.cc:1467] Processing CreateTablet for tablet b9199b0d2b924a79938a979df4196bb4 (DEFAULT_TABLE table=default.Table_44769a7c8b7e463dbe5d907dacc0410f [id=472b3381fca44299922af20255b6aad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.648532 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9199b0d2b924a79938a979df4196bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.651547 1177 tablet_bootstrap.cc:654] T 6834be02864543469056a928fafe28ca P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.652405 759 tablet_service.cc:1467] Processing CreateTablet for tablet b9199b0d2b924a79938a979df4196bb4 (DEFAULT_TABLE table=default.Table_44769a7c8b7e463dbe5d907dacc0410f [id=472b3381fca44299922af20255b6aad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.652813 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9199b0d2b924a79938a979df4196bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.653035 760 tablet_service.cc:1467] Processing CreateTablet for tablet 42cb7e7a41634cf1bdb978ac9d2befcd (DEFAULT_TABLE table=default.Table_3530ac60f1e04607aafce2de389968df [id=8245f0e407da4a88ad69764267feed0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.653388 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42cb7e7a41634cf1bdb978ac9d2befcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.658987 1175 ts_tablet_manager.cc:1918] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.659113 1175 log.cc:1198] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/325e43fdf2fc4eacb48a5227e2ff3bf7
I20250112 02:52:24.659451 1175 ts_tablet_manager.cc:1939] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:24.660554 899 tablet_service.cc:1467] Processing CreateTablet for tablet 42cb7e7a41634cf1bdb978ac9d2befcd (DEFAULT_TABLE table=default.Table_3530ac60f1e04607aafce2de389968df [id=8245f0e407da4a88ad69764267feed0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.660881 754 tablet_service.cc:1467] Processing CreateTablet for tablet aadb2ab1d2024f889841ac91e065e064 (DEFAULT_TABLE table=default.Table_6255caf5a65d4d9192e68f39413398f0 [id=5e4665095f4947bd8adb7ec66be4b456]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.660964 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42cb7e7a41634cf1bdb978ac9d2befcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.661255 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aadb2ab1d2024f889841ac91e065e064. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.661618 1173 tablet_bootstrap.cc:492] T 3f5b418c17084420b867498fda79fb0a P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:24.661780 1173 ts_tablet_manager.cc:1397] T 3f5b418c17084420b867498fda79fb0a P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.108s user 0.001s sys 0.002s
I20250112 02:52:24.662449 1173 raft_consensus.cc:357] T 3f5b418c17084420b867498fda79fb0a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.662652 1173 raft_consensus.cc:383] T 3f5b418c17084420b867498fda79fb0a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.662772 1173 raft_consensus.cc:738] T 3f5b418c17084420b867498fda79fb0a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.663023 1173 consensus_queue.cc:260] T 3f5b418c17084420b867498fda79fb0a P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.663589 1173 ts_tablet_manager.cc:1428] T 3f5b418c17084420b867498fda79fb0a P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:24.663918 1173 tablet_bootstrap.cc:492] T 11afb91c4fde4f6c8204c1cae22cffc5 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:24.671061 1177 tablet_bootstrap.cc:492] T 6834be02864543469056a928fafe28ca P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:24.671257 1177 ts_tablet_manager.cc:1397] T 6834be02864543469056a928fafe28ca P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.003s
I20250112 02:52:24.672175 526 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3f5b418c17084420b867498fda79fb0a (table default.Table_0b78c0ecb1f245838a0117009873c0e9 [id=ad0be37e6b6f4b9597dcb4cbd50d5b12]). Expected version 1 got 0
I20250112 02:52:24.674506 1177 raft_consensus.cc:357] T 6834be02864543469056a928fafe28ca P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.674719 1177 raft_consensus.cc:383] T 6834be02864543469056a928fafe28ca P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.674789 1177 raft_consensus.cc:738] T 6834be02864543469056a928fafe28ca P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.675556 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 (table default.Table_f7f876c63e68490c805c0bfa3f0c25ea [id=5c4769bfcefd4fe0a0f3c2e315d50c32]) successfully deleted
I20250112 02:52:24.676673 1171 tablet_replica.cc:331] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:24.677189 1175 tablet_replica.cc:331] T e8220011788a480eb50ee4b670c45739 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:24.677390 1175 raft_consensus.cc:2238] T e8220011788a480eb50ee4b670c45739 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.677448 526 master_service.cc:946] ReplaceTablet: received request to replace tablet 2bca7a6cecdc49aeb9aa4ae65beb7757 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:24.677450 1171 raft_consensus.cc:2238] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.677549 1175 raft_consensus.cc:2267] T e8220011788a480eb50ee4b670c45739 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.677687 1171 raft_consensus.cc:2267] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.677991 1177 consensus_queue.cc:260] T 6834be02864543469056a928fafe28ca P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.678570 896 tablet_service.cc:1467] Processing CreateTablet for tablet aadb2ab1d2024f889841ac91e065e064 (DEFAULT_TABLE table=default.Table_6255caf5a65d4d9192e68f39413398f0 [id=5e4665095f4947bd8adb7ec66be4b456]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.678800 1175 ts_tablet_manager.cc:1905] T e8220011788a480eb50ee4b670c45739 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.678926 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aadb2ab1d2024f889841ac91e065e064. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.679520 1177 ts_tablet_manager.cc:1428] T 6834be02864543469056a928fafe28ca P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250112 02:52:24.679869 1177 tablet_bootstrap.cc:492] T 2728a022ef0d4952966d397a32356503 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:24.682056 1259 tablet.cc:1721] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:24.684919 759 tablet_service.cc:1467] Processing CreateTablet for tablet 3c686fdc8896463c81c9f7c5f53a3da2 (DEFAULT_TABLE table=default.Table_f7f876c63e68490c805c0bfa3f0c25ea [id=5c4769bfcefd4fe0a0f3c2e315d50c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.687753 893 tablet_service.cc:1467] Processing CreateTablet for tablet 3c686fdc8896463c81c9f7c5f53a3da2 (DEFAULT_TABLE table=default.Table_f7f876c63e68490c805c0bfa3f0c25ea [id=5c4769bfcefd4fe0a0f3c2e315d50c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.688062 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c686fdc8896463c81c9f7c5f53a3da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.688098 519 master_service.cc:946] ReplaceTablet: received request to replace tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:24.688832 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 99715f37fbef47c6860fa1d95e350f20
I20250112 02:52:24.689548 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 42cb7e7a41634cf1bdb978ac9d2befcd
I20250112 02:52:24.689893 1028 tablet_service.cc:1467] Processing CreateTablet for tablet aadb2ab1d2024f889841ac91e065e064 (DEFAULT_TABLE table=default.Table_6255caf5a65d4d9192e68f39413398f0 [id=5e4665095f4947bd8adb7ec66be4b456]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.691744 517 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3f5b418c17084420b867498fda79fb0a (table default.Table_0b78c0ecb1f245838a0117009873c0e9 [id=ad0be37e6b6f4b9597dcb4cbd50d5b12]). Expected version 1 got 0
I20250112 02:52:24.692000 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_893bf955f62b495c9dcc59390b472f96" } modify_external_catalogs: true
I20250112 02:52:24.692203 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_893bf955f62b495c9dcc59390b472f96" } modify_external_catalogs: true
I20250112 02:52:24.693584 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 3c686fdc8896463c81c9f7c5f53a3da2 (DEFAULT_TABLE table=default.Table_f7f876c63e68490c805c0bfa3f0c25ea [id=5c4769bfcefd4fe0a0f3c2e315d50c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.693957 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c686fdc8896463c81c9f7c5f53a3da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.696379 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 99715f37fbef47c6860fa1d95e350f20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.697275 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 42cb7e7a41634cf1bdb978ac9d2befcd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.699081 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet a47cdc208fbc431b8326364e84432cf5
I20250112 02:52:24.699950 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet aadb2ab1d2024f889841ac91e065e064
I20250112 02:52:24.700536 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet f9186ac46ed441ee9308490e90da0e23
I20250112 02:52:24.700948 520 catalog_manager.cc:6596] ReplaceTablet: tablet f9186ac46ed441ee9308490e90da0e23 deleted and replaced by tablet 04f135b226774285a077e5806131d91b
I20250112 02:52:24.701656 894 tablet_service.cc:1514] Processing DeleteTablet for tablet aadb2ab1d2024f889841ac91e065e064 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.701936 888 tablet_service.cc:1514] Processing DeleteTablet for tablet f9186ac46ed441ee9308490e90da0e23 with delete_type TABLET_DATA_DELETED (replaced by tablet 04f135b226774285a077e5806131d91b) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.702147 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 99715f37fbef47c6860fa1d95e350f20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.702199 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet f9186ac46ed441ee9308490e90da0e23 with delete_type TABLET_DATA_DELETED (replaced by tablet 04f135b226774285a077e5806131d91b) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.702373 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet a47cdc208fbc431b8326364e84432cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.702456 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 42cb7e7a41634cf1bdb978ac9d2befcd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.702670 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet aadb2ab1d2024f889841ac91e065e064 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.703163 1181 tablet_bootstrap.cc:492] T aa67bf25dc894d7ea96d247a22c14e03 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:24.703356 1181 ts_tablet_manager.cc:1397] T aa67bf25dc894d7ea96d247a22c14e03 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.001s
I20250112 02:52:24.703931 524 catalog_manager.cc:6071] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: default.Table_0b78c0ecb1f245838a0117009873c0e9 [id=ad0be37e6b6f4b9597dcb4cbd50d5b12] alter complete (version 1)
I20250112 02:52:24.704310 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet b7bd96fb381f4e73a0081be3fc799c36
I20250112 02:52:24.705901 1181 raft_consensus.cc:357] T aa67bf25dc894d7ea96d247a22c14e03 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.706180 1181 raft_consensus.cc:383] T aa67bf25dc894d7ea96d247a22c14e03 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.706174 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet b7bd96fb381f4e73a0081be3fc799c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.706301 1181 raft_consensus.cc:738] T aa67bf25dc894d7ea96d247a22c14e03 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.706607 1181 consensus_queue.cc:260] T aa67bf25dc894d7ea96d247a22c14e03 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.707942 1181 ts_tablet_manager.cc:1428] T aa67bf25dc894d7ea96d247a22c14e03 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:24.708842 1181 tablet_bootstrap.cc:492] T 09886cd08c714f24a96e4df02bc0e6c8 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:24.709667 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_d327837707034a8297f472fd6a4f7af2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.710237 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d327837707034a8297f472fd6a4f7af2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.710078 896 tablet_service.cc:1514] Processing DeleteTablet for tablet a47cdc208fbc431b8326364e84432cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.710687 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 235e9dd0964c4ea6af7d05ef58acae0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.710561 1260 tablet.cc:1721] T 3f5b418c17084420b867498fda79fb0a P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:24.711444 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_47f5f06ece8945aba6226a9c4c928e66" } new_table_name: "default.Table_aa2df7ab46bb4ba5b5e774fe121c371b" modify_external_catalogs: true
I20250112 02:52:24.712749 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_24713b20c8e044f58b27831ee0f2580c" } new_table_name: "default.Table_f4a8edd842744e31baa73dbe3db05c75" modify_external_catalogs: true
I20250112 02:52:24.713537 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c686fdc8896463c81c9f7c5f53a3da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.713881 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_ea7aa744f34e4d5ebcffb74ae74533d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.714358 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea7aa744f34e4d5ebcffb74ae74533d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.716586 517 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3f5b418c17084420b867498fda79fb0a (table default.Table_0b78c0ecb1f245838a0117009873c0e9 [id=ad0be37e6b6f4b9597dcb4cbd50d5b12]). Expected version 1 got 0
I20250112 02:52:24.720091 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_c26869f6cbd540fe980dcf207b5c729d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.720563 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c26869f6cbd540fe980dcf207b5c729d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.687182 1171 ts_tablet_manager.cc:1905] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.690834 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 6834be02864543469056a928fafe28ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.691164 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 42cb7e7a41634cf1bdb978ac9d2befcd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.691205 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 99715f37fbef47c6860fa1d95e350f20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.703374 757 tablet_service.cc:1514] Processing DeleteTablet for tablet a47cdc208fbc431b8326364e84432cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.703612 756 tablet_service.cc:1514] Processing DeleteTablet for tablet aadb2ab1d2024f889841ac91e065e064 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.703641 751 tablet_service.cc:1514] Processing DeleteTablet for tablet f9186ac46ed441ee9308490e90da0e23 with delete_type TABLET_DATA_DELETED (replaced by tablet 04f135b226774285a077e5806131d91b) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.704958 758 tablet_service.cc:1514] Processing DeleteTablet for tablet b7bd96fb381f4e73a0081be3fc799c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.721204 899 tablet_service.cc:1514] Processing DeleteTablet for tablet aa67bf25dc894d7ea96d247a22c14e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.719406 893 tablet_service.cc:1514] Processing DeleteTablet for tablet b7bd96fb381f4e73a0081be3fc799c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.722254 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aadb2ab1d2024f889841ac91e065e064. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:24.727919 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09d06e2e8c13464db8cf801a292bdc99 consensus state has no leader
W20250112 02:52:24.728652 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb724feccc7f4fbc846fb6fa61be3385 consensus state has no leader
I20250112 02:52:24.727577 1253 mvcc.cc:204] Tried to move back new op lower bound from 7113319811545268224 to 7113319811191287808. Current Snapshot: MvccSnapshot[applied={T|T < 7113319811545268224}]
I20250112 02:52:24.733130 1030 tablet_service.cc:1467] Processing CreateTablet for tablet cc62dcbf67e84e36ae1fbe127ccd4879 (DEFAULT_TABLE table=default.Table_fd0169db0f984d3cbc0d8624dda686bc [id=9e09b6670c6e4e57b62f31fd4755c18b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.733494 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc62dcbf67e84e36ae1fbe127ccd4879. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.733697 758 tablet_service.cc:1467] Processing CreateTablet for tablet cc62dcbf67e84e36ae1fbe127ccd4879 (DEFAULT_TABLE table=default.Table_fd0169db0f984d3cbc0d8624dda686bc [id=9e09b6670c6e4e57b62f31fd4755c18b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.734841 751 tablet_service.cc:1467] Processing CreateTablet for tablet 55fb1a197d7e40689ccad82a44492371 (DEFAULT_TABLE table=default.Table_f3695b00a89b4353903587a6c1418d8c [id=d1781738b8a74f139b09863918cbb21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.735440 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 55fb1a197d7e40689ccad82a44492371 (DEFAULT_TABLE table=default.Table_f3695b00a89b4353903587a6c1418d8c [id=d1781738b8a74f139b09863918cbb21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.736088 756 tablet_service.cc:1467] Processing CreateTablet for tablet 41e166b3f8414ababbd4a3f07fb90c4c (DEFAULT_TABLE table=default.Table_11b4065cfb0449b68950d63e65f566a2 [id=0e325561bbdb468e8a64ca9808e2f3cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.736683 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69
I20250112 02:52:24.737080 519 catalog_manager.cc:6596] ReplaceTablet: tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 deleted and replaced by tablet 1c639a2bcd4740e4be7b2b162004669a
I20250112 02:52:24.737143 757 tablet_service.cc:1467] Processing CreateTablet for tablet 23e7a2e88c374e949ced92c557225cbf (DEFAULT_TABLE table=default.Table_952149cc55fa44fdbae173c5e83bc4f5 [id=bfa9a504a8d34c86929efb0d15d049d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.737571 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c639a2bcd4740e4be7b2b162004669a) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.737761 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c639a2bcd4740e4be7b2b162004669a) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.738797 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 41e166b3f8414ababbd4a3f07fb90c4c (DEFAULT_TABLE table=default.Table_11b4065cfb0449b68950d63e65f566a2 [id=0e325561bbdb468e8a64ca9808e2f3cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.739190 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c639a2bcd4740e4be7b2b162004669a) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.739504 899 tablet_service.cc:1467] Processing CreateTablet for tablet 55fb1a197d7e40689ccad82a44492371 (DEFAULT_TABLE table=default.Table_f3695b00a89b4353903587a6c1418d8c [id=d1781738b8a74f139b09863918cbb21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.739627 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 23e7a2e88c374e949ced92c557225cbf (DEFAULT_TABLE table=default.Table_952149cc55fa44fdbae173c5e83bc4f5 [id=bfa9a504a8d34c86929efb0d15d049d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.739847 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55fb1a197d7e40689ccad82a44492371. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.740391 897 tablet_service.cc:1467] Processing CreateTablet for tablet 23e7a2e88c374e949ced92c557225cbf (DEFAULT_TABLE table=default.Table_952149cc55fa44fdbae173c5e83bc4f5 [id=bfa9a504a8d34c86929efb0d15d049d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.740515 754 tablet_service.cc:1467] Processing CreateTablet for tablet 4916ae3917084cfe8c93240a8857be0e (DEFAULT_TABLE table=default.Table_32538719e60a4843b560a8a6f3ac17b1 [id=c7052afaf6284ef0972835c1f46cb829]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.740705 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23e7a2e88c374e949ced92c557225cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.741662 893 tablet_service.cc:1467] Processing CreateTablet for tablet cc62dcbf67e84e36ae1fbe127ccd4879 (DEFAULT_TABLE table=default.Table_fd0169db0f984d3cbc0d8624dda686bc [id=9e09b6670c6e4e57b62f31fd4755c18b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.742286 889 tablet_service.cc:1467] Processing CreateTablet for tablet 4916ae3917084cfe8c93240a8857be0e (DEFAULT_TABLE table=default.Table_32538719e60a4843b560a8a6f3ac17b1 [id=c7052afaf6284ef0972835c1f46cb829]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.742592 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4916ae3917084cfe8c93240a8857be0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.743623 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 4916ae3917084cfe8c93240a8857be0e (DEFAULT_TABLE table=default.Table_32538719e60a4843b560a8a6f3ac17b1 [id=c7052afaf6284ef0972835c1f46cb829]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.746031 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55fb1a197d7e40689ccad82a44492371. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.746114 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41e166b3f8414ababbd4a3f07fb90c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.746306 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23e7a2e88c374e949ced92c557225cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.746501 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23e7a2e88c374e949ced92c557225cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.751631 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc62dcbf67e84e36ae1fbe127ccd4879. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.751667 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4916ae3917084cfe8c93240a8857be0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.751818 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55fb1a197d7e40689ccad82a44492371. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.752090 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41e166b3f8414ababbd4a3f07fb90c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.755718 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4916ae3917084cfe8c93240a8857be0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.758608 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_f4d612cd6a524bc493147493c5a4a284"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.759105 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4d612cd6a524bc493147493c5a4a284 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.762612 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_f4a8edd842744e31baa73dbe3db05c75" } new_table_name: "default.Table_91be8f5d6eb24e2ea86d927b7f062d1e" modify_external_catalogs: true
I20250112 02:52:24.763929 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 42cb7e7a41634cf1bdb978ac9d2befcd (DEFAULT_TABLE table=default.Table_3530ac60f1e04607aafce2de389968df [id=8245f0e407da4a88ad69764267feed0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.764698 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42cb7e7a41634cf1bdb978ac9d2befcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.773100 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_83a9b08ba3c64e50bbb471dbe4a7ad70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.773874 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83a9b08ba3c64e50bbb471dbe4a7ad70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.781131 524 master_service.cc:946] ReplaceTablet: received request to replace tablet 6834be02864543469056a928fafe28ca from {username='slave'} at 127.0.0.1:40934
W20250112 02:52:24.783296 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet a1a0a13696fc41ccb28218f1275a4b5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1a0a13696fc41ccb28218f1275a4b5d already in progress: creating tablet
I20250112 02:52:24.785503 1177 tablet_bootstrap.cc:654] T 2728a022ef0d4952966d397a32356503 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.788184 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_10e019e7a9244dbeac8bd2a55f618830"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.788728 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10e019e7a9244dbeac8bd2a55f618830 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:24.794706 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 already in progress: creating tablet
W20250112 02:52:24.795563 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet aa67bf25dc894d7ea96d247a22c14e03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa67bf25dc894d7ea96d247a22c14e03 already in progress: creating tablet
W20250112 02:52:24.796219 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 2b34fb96bc5140118725af8346ea827f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b34fb96bc5140118725af8346ea827f already in progress: creating tablet
W20250112 02:52:24.796803 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet aba1744138fe42ae9d097db01cac083d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aba1744138fe42ae9d097db01cac083d already in progress: creating tablet
I20250112 02:52:24.797138 1172 tablet_replica.cc:331] T d02caa04543d43e086f5cd56ac1ead1a P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:24.797374 1172 raft_consensus.cc:2238] T d02caa04543d43e086f5cd56ac1ead1a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.797600 1172 raft_consensus.cc:2267] T d02caa04543d43e086f5cd56ac1ead1a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.800168 1173 tablet_bootstrap.cc:654] T 11afb91c4fde4f6c8204c1cae22cffc5 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.801647 1173 tablet_bootstrap.cc:492] T 11afb91c4fde4f6c8204c1cae22cffc5 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:24.801807 1173 ts_tablet_manager.cc:1397] T 11afb91c4fde4f6c8204c1cae22cffc5 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.138s user 0.004s sys 0.000s
I20250112 02:52:24.802471 1173 raft_consensus.cc:357] T 11afb91c4fde4f6c8204c1cae22cffc5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.802690 1173 raft_consensus.cc:383] T 11afb91c4fde4f6c8204c1cae22cffc5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.802758 1173 raft_consensus.cc:738] T 11afb91c4fde4f6c8204c1cae22cffc5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.803026 1173 consensus_queue.cc:260] T 11afb91c4fde4f6c8204c1cae22cffc5 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.803490 1173 ts_tablet_manager.cc:1428] T 11afb91c4fde4f6c8204c1cae22cffc5 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:24.804842 896 tablet_service.cc:1467] Processing CreateTablet for tablet 41e166b3f8414ababbd4a3f07fb90c4c (DEFAULT_TABLE table=default.Table_11b4065cfb0449b68950d63e65f566a2 [id=0e325561bbdb468e8a64ca9808e2f3cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.804318 1173 tablet_bootstrap.cc:492] T bb0030637279449fa9c00cdc075fbca9 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:24.807968 1171 ts_tablet_manager.cc:1918] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.808111 1171 log.cc:1198] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/325e43fdf2fc4eacb48a5227e2ff3bf7
I20250112 02:52:24.808429 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 11afb91c4fde4f6c8204c1cae22cffc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.808471 1171 ts_tablet_manager.cc:1939] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:24.809131 1173 tablet_bootstrap.cc:654] T bb0030637279449fa9c00cdc075fbca9 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.809649 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 (table default.Table_f7f876c63e68490c805c0bfa3f0c25ea [id=5c4769bfcefd4fe0a0f3c2e315d50c32]) successfully deleted
I20250112 02:52:24.810043 1177 tablet_bootstrap.cc:492] T 2728a022ef0d4952966d397a32356503 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:24.810168 1177 ts_tablet_manager.cc:1397] T 2728a022ef0d4952966d397a32356503 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.130s user 0.003s sys 0.000s
I20250112 02:52:24.810869 1177 raft_consensus.cc:357] T 2728a022ef0d4952966d397a32356503 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.811040 1177 raft_consensus.cc:383] T 2728a022ef0d4952966d397a32356503 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.811113 1177 raft_consensus.cc:738] T 2728a022ef0d4952966d397a32356503 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
W20250112 02:52:24.811372 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb724feccc7f4fbc846fb6fa61be3385 consensus state has no leader
I20250112 02:52:24.811306 1177 consensus_queue.cc:260] T 2728a022ef0d4952966d397a32356503 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.811961 1177 ts_tablet_manager.cc:1428] T 2728a022ef0d4952966d397a32356503 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:24.812208 1177 tablet_bootstrap.cc:492] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:24.814301 1172 ts_tablet_manager.cc:1905] T d02caa04543d43e086f5cd56ac1ead1a P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.815335 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 2728a022ef0d4952966d397a32356503 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.816520 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_f834660b442848b3b36f079110711bb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:24.816706 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_65f28b8b506644058434fb4736a80acc" } new_table_name: "default.Table_4f66980b946046a4bd2531c4ab094352" modify_external_catalogs: true
W20250112 02:52:24.816989 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f834660b442848b3b36f079110711bb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.817175 1171 tablet_replica.cc:331] T 6834be02864543469056a928fafe28ca P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:24.817384 1177 tablet_bootstrap.cc:654] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.817932 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41e166b3f8414ababbd4a3f07fb90c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.818248 1171 raft_consensus.cc:2238] T 6834be02864543469056a928fafe28ca P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.818398 1171 raft_consensus.cc:2267] T 6834be02864543469056a928fafe28ca P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.819160 1171 ts_tablet_manager.cc:1905] T 6834be02864543469056a928fafe28ca P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.819547 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc62dcbf67e84e36ae1fbe127ccd4879. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.822722 1172 ts_tablet_manager.cc:1918] T d02caa04543d43e086f5cd56ac1ead1a P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.822853 1172 log.cc:1198] T d02caa04543d43e086f5cd56ac1ead1a P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/d02caa04543d43e086f5cd56ac1ead1a
I20250112 02:52:24.823205 1172 ts_tablet_manager.cc:1939] T d02caa04543d43e086f5cd56ac1ead1a P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:24.823951 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_096703f878244b8dac2dcf187fd23fd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:24.824291 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet d02caa04543d43e086f5cd56ac1ead1a (table default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94]) successfully deleted
W20250112 02:52:24.824409 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_096703f878244b8dac2dcf187fd23fd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:24.824837 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 6b521522c4784d2ab4056268823f1099 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b521522c4784d2ab4056268823f1099 already in progress: creating tablet
I20250112 02:52:24.825449 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_2ce39f46be3243acb7f7f99a0e9ac9fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.825923 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ce39f46be3243acb7f7f99a0e9ac9fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.826310 1181 tablet_bootstrap.cc:654] T 09886cd08c714f24a96e4df02bc0e6c8 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:24.826648 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet d3e41e307df64109a41b0c1c44d3bda6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3e41e307df64109a41b0c1c44d3bda6 already in progress: creating tablet
I20250112 02:52:24.827299 1172 tablet_replica.cc:331] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:24.827445 1172 raft_consensus.cc:2238] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.827625 1172 raft_consensus.cc:2267] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.827852 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_b29145eb86ed42eabdd10eb4a9026639" } new_table_name: "default.Table_96a35d4e94284a49ba1a88d7d0acd3c1" modify_external_catalogs: true
I20250112 02:52:24.828346 1172 ts_tablet_manager.cc:1905] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.828563 1171 ts_tablet_manager.cc:1918] T 6834be02864543469056a928fafe28ca P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.828680 1171 log.cc:1198] T 6834be02864543469056a928fafe28ca P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/6834be02864543469056a928fafe28ca
I20250112 02:52:24.829054 1171 ts_tablet_manager.cc:1939] T 6834be02864543469056a928fafe28ca P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:24.829013 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_981a70d4d3184fd8a99e6b81ae1caab7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.829437 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_981a70d4d3184fd8a99e6b81ae1caab7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.830000 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 6834be02864543469056a928fafe28ca (table default.Table_5fe703bbcb87402b87a82e1441569d53 [id=a2449caa011d43bfb54aea196629c9f1]) successfully deleted
W20250112 02:52:24.830566 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 42cb7e7a41634cf1bdb978ac9d2befcd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42cb7e7a41634cf1bdb978ac9d2befcd already in progress: creating tablet
W20250112 02:52:24.831079 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 99715f37fbef47c6860fa1d95e350f20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99715f37fbef47c6860fa1d95e350f20 already in progress: creating tablet
W20250112 02:52:24.831971 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet a47cdc208fbc431b8326364e84432cf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a47cdc208fbc431b8326364e84432cf5 already in progress: creating tablet
I20250112 02:52:24.832540 1177 tablet_bootstrap.cc:492] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:24.832665 1177 ts_tablet_manager.cc:1397] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20250112 02:52:24.833240 1177 raft_consensus.cc:357] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:24.833417 1177 raft_consensus.cc:383] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.833477 1177 raft_consensus.cc:738] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.833714 1177 consensus_queue.cc:260] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:24.835006 759 tablet_service.cc:1467] Processing CreateTablet for tablet f8c01b80fc8242d98fdfbf767212144d (DEFAULT_TABLE table=default.Table_d327837707034a8297f472fd6a4f7af2 [id=9807191b6e5c4d478cd47a54464fb9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.835328 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8c01b80fc8242d98fdfbf767212144d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:24.838045 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 54033a747d5a49b0a1c16f138158f596 consensus state has no leader
W20250112 02:52:24.839190 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c2595782d66e4e4b890c94221b196daa consensus state has no leader
I20250112 02:52:24.840051 1177 ts_tablet_manager.cc:1428] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250112 02:52:24.840312 1177 tablet_bootstrap.cc:492] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:24.841764 518 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5a8e1b96accd459eaed221ded4a453b6 (table default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94]). Expected version 2 got 0
I20250112 02:52:24.841953 754 tablet_service.cc:1467] Processing CreateTablet for tablet 64707ee432fb4e96a3bbf7252dfa1813 (DEFAULT_TABLE table=default.Table_c26869f6cbd540fe980dcf207b5c729d [id=cdd6a2097792484d84bca856f30d7ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.842267 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64707ee432fb4e96a3bbf7252dfa1813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.843778 896 tablet_service.cc:1467] Processing CreateTablet for tablet f8c01b80fc8242d98fdfbf767212144d (DEFAULT_TABLE table=default.Table_d327837707034a8297f472fd6a4f7af2 [id=9807191b6e5c4d478cd47a54464fb9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.844292 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8c01b80fc8242d98fdfbf767212144d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.846107 1034 tablet_service.cc:1467] Processing CreateTablet for tablet f8c01b80fc8242d98fdfbf767212144d (DEFAULT_TABLE table=default.Table_d327837707034a8297f472fd6a4f7af2 [id=9807191b6e5c4d478cd47a54464fb9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.846487 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8c01b80fc8242d98fdfbf767212144d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.851402 1190 raft_consensus.cc:491] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:24.851563 1190 raft_consensus.cc:513] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:24.851989 751 tablet_service.cc:1467] Processing CreateTablet for tablet 04f135b226774285a077e5806131d91b (DEFAULT_TABLE table=default.Table_17d23578eb794e99829a550bed4a11d9 [id=f93b5bb22ec1424ea047383ed9713618]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.852200 1190 leader_election.cc:290] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:24.852304 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04f135b226774285a077e5806131d91b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.852979 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8e1b96accd459eaed221ded4a453b6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:24.853508 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8e1b96accd459eaed221ded4a453b6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:24.853740 780 raft_consensus.cc:2463] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
W20250112 02:52:24.854131 854 leader_election.cc:343] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:24.854552 855 leader_election.cc:304] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:24.854984 1190 raft_consensus.cc:2798] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:24.855098 1190 raft_consensus.cc:491] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:24.855175 1190 raft_consensus.cc:3054] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:24.856230 1172 ts_tablet_manager.cc:1918] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.856355 1172 log.cc:1198] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/2bca7a6cecdc49aeb9aa4ae65beb7757
I20250112 02:52:24.856776 1172 ts_tablet_manager.cc:1939] T 2bca7a6cecdc49aeb9aa4ae65beb7757 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:24.857434 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 2bca7a6cecdc49aeb9aa4ae65beb7757 (table default.Table_8db6a50cdd8a4fb9a4276fd240476bcd [id=01845880c50e41378a0c744d3b67cdbf]) successfully deleted
I20250112 02:52:24.857470 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 1c639a2bcd4740e4be7b2b162004669a (DEFAULT_TABLE table=default.Table_f7982d4045f04157ae3aa0e8e8f6ffda [id=2f43ebd749304c0287769224ac6b12c7]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:24.857681 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b885247d2cb449e0a29f6f5d81518589 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b885247d2cb449e0a29f6f5d81518589 already in progress: creating tablet
I20250112 02:52:24.857843 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c639a2bcd4740e4be7b2b162004669a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:24.857905 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 71b5f80ed198491780cbe17e6cd3fa15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71b5f80ed198491780cbe17e6cd3fa15 already in progress: creating tablet
W20250112 02:52:24.858094 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 49650a419dfd49d5a1547770d3f2fe54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49650a419dfd49d5a1547770d3f2fe54 already in progress: creating tablet
W20250112 02:52:24.858333 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b2d8ea2ef18a46508371b4789ab6f4fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2d8ea2ef18a46508371b4789ab6f4fa already in progress: creating tablet
I20250112 02:52:24.858503 1172 tablet_replica.cc:331] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
W20250112 02:52:24.858547 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 already in progress: creating tablet
I20250112 02:52:24.858688 1172 raft_consensus.cc:2238] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.858860 1172 raft_consensus.cc:2267] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.859484 1172 ts_tablet_manager.cc:1905] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.859642 756 tablet_service.cc:1467] Processing CreateTablet for tablet 1c639a2bcd4740e4be7b2b162004669a (DEFAULT_TABLE table=default.Table_f7982d4045f04157ae3aa0e8e8f6ffda [id=2f43ebd749304c0287769224ac6b12c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.861639 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c639a2bcd4740e4be7b2b162004669a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.861917 899 tablet_service.cc:1467] Processing CreateTablet for tablet 1c639a2bcd4740e4be7b2b162004669a (DEFAULT_TABLE table=default.Table_f7982d4045f04157ae3aa0e8e8f6ffda [id=2f43ebd749304c0287769224ac6b12c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.862246 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c639a2bcd4740e4be7b2b162004669a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.865427 1172 ts_tablet_manager.cc:1918] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.865558 1172 log.cc:1198] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/e56a00ec3f5040b982c555d65f07d34d
I20250112 02:52:24.865957 1172 ts_tablet_manager.cc:1939] T e56a00ec3f5040b982c555d65f07d34d P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:24.868042 1175 ts_tablet_manager.cc:1918] T e8220011788a480eb50ee4b670c45739 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.868182 1175 log.cc:1198] T e8220011788a480eb50ee4b670c45739 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/e8220011788a480eb50ee4b670c45739
I20250112 02:52:24.868505 1175 ts_tablet_manager.cc:1939] T e8220011788a480eb50ee4b670c45739 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:24.868836 1172 tablet_replica.cc:331] T 11afb91c4fde4f6c8204c1cae22cffc5 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:24.868992 1172 raft_consensus.cc:2238] T 11afb91c4fde4f6c8204c1cae22cffc5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.869154 1172 raft_consensus.cc:2267] T 11afb91c4fde4f6c8204c1cae22cffc5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.869191 889 tablet_service.cc:1467] Processing CreateTablet for tablet 04f135b226774285a077e5806131d91b (DEFAULT_TABLE table=default.Table_17d23578eb794e99829a550bed4a11d9 [id=f93b5bb22ec1424ea047383ed9713618]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.869549 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04f135b226774285a077e5806131d91b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.869534 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet e8220011788a480eb50ee4b670c45739 (table default.Table_edaa8b50dfa8463e99e05ab8d1d89bc3 [id=a6a18be9a78e4959b7d272c447f6fc25]) successfully deleted
W20250112 02:52:24.870349 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet aadb2ab1d2024f889841ac91e065e064 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aadb2ab1d2024f889841ac91e065e064 already in progress: creating tablet
W20250112 02:52:24.870568 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet f9186ac46ed441ee9308490e90da0e23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9186ac46ed441ee9308490e90da0e23 already in progress: creating tablet
W20250112 02:52:24.870767 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 99715f37fbef47c6860fa1d95e350f20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99715f37fbef47c6860fa1d95e350f20 already in progress: creating tablet
W20250112 02:52:24.870978 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 42cb7e7a41634cf1bdb978ac9d2befcd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42cb7e7a41634cf1bdb978ac9d2befcd already in progress: creating tablet
I20250112 02:52:24.871137 1175 tablet_replica.cc:331] T 235e9dd0964c4ea6af7d05ef58acae0d P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
W20250112 02:52:24.871174 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet a47cdc208fbc431b8326364e84432cf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a47cdc208fbc431b8326364e84432cf5 already in progress: creating tablet
I20250112 02:52:24.871287 1175 raft_consensus.cc:2238] T 235e9dd0964c4ea6af7d05ef58acae0d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.871431 1175 raft_consensus.cc:2267] T 235e9dd0964c4ea6af7d05ef58acae0d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.871677 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet e56a00ec3f5040b982c555d65f07d34d (table default.Table_1ae04ce4866246fc9e5fe0be72adcf64 [id=aa8325dff47c4d95832e0107cd5a17e9]) successfully deleted
I20250112 02:52:24.871912 521 master_service.cc:946] ReplaceTablet: received request to replace tablet f05d3326545849b9af4519bc66c21069 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:24.872201 1175 ts_tablet_manager.cc:1905] T 235e9dd0964c4ea6af7d05ef58acae0d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.872864 1172 ts_tablet_manager.cc:1905] T 11afb91c4fde4f6c8204c1cae22cffc5 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:24.871918 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 99715f37fbef47c6860fa1d95e350f20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99715f37fbef47c6860fa1d95e350f20 already in progress: creating tablet
W20250112 02:52:24.874655 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 42cb7e7a41634cf1bdb978ac9d2befcd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42cb7e7a41634cf1bdb978ac9d2befcd already in progress: creating tablet
W20250112 02:52:24.874863 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f9186ac46ed441ee9308490e90da0e23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9186ac46ed441ee9308490e90da0e23 already in progress: creating tablet
I20250112 02:52:24.874819 897 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd9985cd20148f195a857c478b0755f (DEFAULT_TABLE table=default.Table_ea7aa744f34e4d5ebcffb74ae74533d8 [id=e63daf724d874c89badafb8d59653f74]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:24.875043 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet a47cdc208fbc431b8326364e84432cf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a47cdc208fbc431b8326364e84432cf5 already in progress: creating tablet
I20250112 02:52:24.875154 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cd9985cd20148f195a857c478b0755f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:24.875238 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet aadb2ab1d2024f889841ac91e065e064 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aadb2ab1d2024f889841ac91e065e064 already in progress: creating tablet
W20250112 02:52:24.875435 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b7bd96fb381f4e73a0081be3fc799c36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7bd96fb381f4e73a0081be3fc799c36 already in progress: creating tablet
W20250112 02:52:24.875756 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 already in progress: creating tablet
I20250112 02:52:24.876811 1177 tablet_bootstrap.cc:654] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.878548 1172 ts_tablet_manager.cc:1918] T 11afb91c4fde4f6c8204c1cae22cffc5 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.878687 1172 log.cc:1198] T 11afb91c4fde4f6c8204c1cae22cffc5 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/11afb91c4fde4f6c8204c1cae22cffc5
I20250112 02:52:24.879024 1172 ts_tablet_manager.cc:1939] T 11afb91c4fde4f6c8204c1cae22cffc5 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:24.879017 1190 raft_consensus.cc:513] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:24.879942 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 11afb91c4fde4f6c8204c1cae22cffc5 (table default.Table_a78a69e0b1914d25a0f0726d75b66731 [id=e47433a989db4945973f25468fd1581b]) successfully deleted
I20250112 02:52:24.880045 754 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd9985cd20148f195a857c478b0755f (DEFAULT_TABLE table=default.Table_ea7aa744f34e4d5ebcffb74ae74533d8 [id=e63daf724d874c89badafb8d59653f74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.880383 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cd9985cd20148f195a857c478b0755f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.880501 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8e1b96accd459eaed221ded4a453b6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:24.880750 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8e1b96accd459eaed221ded4a453b6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:24.880950 780 raft_consensus.cc:3054] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:24.881168 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 04f135b226774285a077e5806131d91b (DEFAULT_TABLE table=default.Table_17d23578eb794e99829a550bed4a11d9 [id=f93b5bb22ec1424ea047383ed9713618]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.881422 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04f135b226774285a077e5806131d91b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:24.883648 854 leader_election.cc:343] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:24.883818 1190 leader_election.cc:290] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:24.883783 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd9985cd20148f195a857c478b0755f (DEFAULT_TABLE table=default.Table_ea7aa744f34e4d5ebcffb74ae74533d8 [id=e63daf724d874c89badafb8d59653f74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.884095 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cd9985cd20148f195a857c478b0755f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.884114 780 raft_consensus.cc:2463] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:24.884490 855 leader_election.cc:304] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:24.884950 1190 raft_consensus.cc:2798] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:24.886993 1173 tablet_bootstrap.cc:492] T bb0030637279449fa9c00cdc075fbca9 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:24.887120 1173 ts_tablet_manager.cc:1397] T bb0030637279449fa9c00cdc075fbca9 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.083s user 0.006s sys 0.002s
I20250112 02:52:24.890425 521 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_82662e02aee0459dad1b621df36f58f0" } modify_external_catalogs: true
I20250112 02:52:24.890632 521 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_82662e02aee0459dad1b621df36f58f0" } modify_external_catalogs: true
I20250112 02:52:24.891645 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_4e7e1a3381d2412987a7061d9030f9f9" } modify_external_catalogs: true
I20250112 02:52:24.891832 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_4e7e1a3381d2412987a7061d9030f9f9" } modify_external_catalogs: true
W20250112 02:52:24.892005 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet aadb2ab1d2024f889841ac91e065e064 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aadb2ab1d2024f889841ac91e065e064 already in progress: creating tablet
W20250112 02:52:24.892616 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f9186ac46ed441ee9308490e90da0e23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9186ac46ed441ee9308490e90da0e23 already in progress: creating tablet
W20250112 02:52:24.893143 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b7bd96fb381f4e73a0081be3fc799c36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7bd96fb381f4e73a0081be3fc799c36 already in progress: creating tablet
I20250112 02:52:24.893296 1171 tablet_replica.cc:331] T 2728a022ef0d4952966d397a32356503 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:24.893446 1171 raft_consensus.cc:2238] T 2728a022ef0d4952966d397a32356503 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250112 02:52:24.893473 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 already in progress: creating tablet
I20250112 02:52:24.893705 1171 raft_consensus.cc:2267] T 2728a022ef0d4952966d397a32356503 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.897465 1171 ts_tablet_manager.cc:1905] T 2728a022ef0d4952966d397a32356503 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.900547 1181 tablet_bootstrap.cc:492] T 09886cd08c714f24a96e4df02bc0e6c8 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:24.900679 1181 ts_tablet_manager.cc:1397] T 09886cd08c714f24a96e4df02bc0e6c8 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.192s user 0.003s sys 0.009s
I20250112 02:52:24.901468 1181 raft_consensus.cc:357] T 09886cd08c714f24a96e4df02bc0e6c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.901716 1181 raft_consensus.cc:383] T 09886cd08c714f24a96e4df02bc0e6c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.902005 1181 raft_consensus.cc:738] T 09886cd08c714f24a96e4df02bc0e6c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.902287 1181 consensus_queue.cc:260] T 09886cd08c714f24a96e4df02bc0e6c8 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.904342 893 tablet_service.cc:1467] Processing CreateTablet for tablet 64707ee432fb4e96a3bbf7252dfa1813 (DEFAULT_TABLE table=default.Table_c26869f6cbd540fe980dcf207b5c729d [id=cdd6a2097792484d84bca856f30d7ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.904892 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 64707ee432fb4e96a3bbf7252dfa1813 (DEFAULT_TABLE table=default.Table_c26869f6cbd540fe980dcf207b5c729d [id=cdd6a2097792484d84bca856f30d7ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.907171 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64707ee432fb4e96a3bbf7252dfa1813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.908178 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_091c87de29164fc784158659eeaf545a" } modify_external_catalogs: true
I20250112 02:52:24.908362 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_091c87de29164fc784158659eeaf545a" } modify_external_catalogs: true
I20250112 02:52:24.908411 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_e21cbee8871f4ab583b22a3eaa4b5e6a" } modify_external_catalogs: true
I20250112 02:52:24.908578 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_e21cbee8871f4ab583b22a3eaa4b5e6a" } modify_external_catalogs: true
I20250112 02:52:24.909013 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_11b4065cfb0449b68950d63e65f566a2" } modify_external_catalogs: true
I20250112 02:52:24.909107 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_11b4065cfb0449b68950d63e65f566a2" } modify_external_catalogs: true
I20250112 02:52:24.910046 1175 ts_tablet_manager.cc:1918] T 235e9dd0964c4ea6af7d05ef58acae0d P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.910154 1175 log.cc:1198] T 235e9dd0964c4ea6af7d05ef58acae0d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/235e9dd0964c4ea6af7d05ef58acae0d
I20250112 02:52:24.914278 1175 ts_tablet_manager.cc:1939] T 235e9dd0964c4ea6af7d05ef58acae0d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:24.915406 1190 raft_consensus.cc:695] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:24.916004 1181 ts_tablet_manager.cc:1428] T 09886cd08c714f24a96e4df02bc0e6c8 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.015s user 0.003s sys 0.001s
I20250112 02:52:24.916296 1181 tablet_bootstrap.cc:492] T f05d3326545849b9af4519bc66c21069 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:24.916510 1190 consensus_queue.cc:237] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:24.919303 1181 tablet_bootstrap.cc:654] T f05d3326545849b9af4519bc66c21069 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.920559 1171 ts_tablet_manager.cc:1918] T 2728a022ef0d4952966d397a32356503 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:24.920667 1171 log.cc:1198] T 2728a022ef0d4952966d397a32356503 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/2728a022ef0d4952966d397a32356503
I20250112 02:52:24.921010 1171 ts_tablet_manager.cc:1939] T 2728a022ef0d4952966d397a32356503 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:24.921666 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 235e9dd0964c4ea6af7d05ef58acae0d (table default.Table_ae0485ac89ea47e7a8c19a07c5a63730 [id=e659e0cc87684ca79fc3b8241116742a]) successfully deleted
I20250112 02:52:24.921891 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 2728a022ef0d4952966d397a32356503 (table default.Table_6d358fc697c74529822f0b2b70321a02 [id=9b36703d61a046799cb709c7618b1fbd]) successfully deleted
I20250112 02:52:24.922463 1177 tablet_bootstrap.cc:492] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:24.922592 1177 ts_tablet_manager.cc:1397] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.082s user 0.002s sys 0.005s
I20250112 02:52:24.923314 1177 raft_consensus.cc:357] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.923501 1177 raft_consensus.cc:383] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.923583 1177 raft_consensus.cc:738] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.923906 1177 consensus_queue.cc:260] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.924690 754 tablet_service.cc:1467] Processing CreateTablet for tablet 161fe26b9c0e4543b7aeb0021506d72a (DEFAULT_TABLE table=default.Table_981a70d4d3184fd8a99e6b81ae1caab7 [id=8016eb490f574ad8a5d3ee598fecaeb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.925084 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 161fe26b9c0e4543b7aeb0021506d72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.926307 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_7d54fa05bdbf4ff3b5cce201222e62de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:24.926654 1177 ts_tablet_manager.cc:1428] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250112 02:52:24.926921 1177 tablet_bootstrap.cc:492] T e8220011788a480eb50ee4b670c45739 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:24.927276 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d54fa05bdbf4ff3b5cce201222e62de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.928368 1177 tablet_bootstrap.cc:654] T e8220011788a480eb50ee4b670c45739 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.926304 520 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c2595782d66e4e4b890c94221b196daa (table default.Table_96a35d4e94284a49ba1a88d7d0acd3c1 [id=8559c454b17b4a00a60689a6a9b19f7d]). Expected version 1 got 0
I20250112 02:52:24.930788 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_2ce39f46be3243acb7f7f99a0e9ac9fb" } new_table_name: "default.Table_427e5903e2b04aeeab11c68a7e318d94" modify_external_catalogs: true
I20250112 02:52:24.932760 523 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c2595782d66e4e4b890c94221b196daa (table default.Table_96a35d4e94284a49ba1a88d7d0acd3c1 [id=8559c454b17b4a00a60689a6a9b19f7d]). Expected version 1 got 0
I20250112 02:52:24.933627 896 tablet_service.cc:1467] Processing CreateTablet for tablet 161fe26b9c0e4543b7aeb0021506d72a (DEFAULT_TABLE table=default.Table_981a70d4d3184fd8a99e6b81ae1caab7 [id=8016eb490f574ad8a5d3ee598fecaeb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.933725 756 tablet_service.cc:1467] Processing CreateTablet for tablet 039a4734358646508093200d59d30013 (DEFAULT_TABLE table=default.Table_f834660b442848b3b36f079110711bb3 [id=f32f4eea5a904798baa70ed435c9bf62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.933988 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 161fe26b9c0e4543b7aeb0021506d72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.934042 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 039a4734358646508093200d59d30013. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.936466 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_7abcf14838c744109a08985ba6ce3e0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:24.936890 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7abcf14838c744109a08985ba6ce3e0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.939745 1175 tablet_replica.cc:331] T aa67bf25dc894d7ea96d247a22c14e03 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:24.939945 1175 raft_consensus.cc:2238] T aa67bf25dc894d7ea96d247a22c14e03 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.940099 1175 raft_consensus.cc:2267] T aa67bf25dc894d7ea96d247a22c14e03 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.940471 1173 raft_consensus.cc:357] T bb0030637279449fa9c00cdc075fbca9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.940699 1173 raft_consensus.cc:383] T bb0030637279449fa9c00cdc075fbca9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.940798 1173 raft_consensus.cc:738] T bb0030637279449fa9c00cdc075fbca9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.941028 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 039a4734358646508093200d59d30013 (DEFAULT_TABLE table=default.Table_f834660b442848b3b36f079110711bb3 [id=f32f4eea5a904798baa70ed435c9bf62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.941059 1173 consensus_queue.cc:260] T bb0030637279449fa9c00cdc075fbca9 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.941541 1173 ts_tablet_manager.cc:1428] T bb0030637279449fa9c00cdc075fbca9 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.054s user 0.000s sys 0.002s
I20250112 02:52:24.941823 1173 tablet_bootstrap.cc:492] T 22750265b522471680835e7f98475e6e P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:24.941938 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 161fe26b9c0e4543b7aeb0021506d72a (DEFAULT_TABLE table=default.Table_981a70d4d3184fd8a99e6b81ae1caab7 [id=8016eb490f574ad8a5d3ee598fecaeb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.942169 897 tablet_service.cc:1467] Processing CreateTablet for tablet 039a4734358646508093200d59d30013 (DEFAULT_TABLE table=default.Table_f834660b442848b3b36f079110711bb3 [id=f32f4eea5a904798baa70ed435c9bf62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.942204 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64707ee432fb4e96a3bbf7252dfa1813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.942307 756 tablet_service.cc:1467] Processing CreateTablet for tablet 015412a5fa304453bd8d1be1c3cd7a46 (DEFAULT_TABLE table=default.Table_83a9b08ba3c64e50bbb471dbe4a7ad70 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.942462 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 039a4734358646508093200d59d30013. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.942600 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 015412a5fa304453bd8d1be1c3cd7a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.942714 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 161fe26b9c0e4543b7aeb0021506d72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.946385 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 039a4734358646508093200d59d30013. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.948698 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 4c0c2fad4f3243238641aae56b305bb5
I20250112 02:52:24.949739 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0c2fad4f3243238641aae56b305bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.949784 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 41e166b3f8414ababbd4a3f07fb90c4c
I20250112 02:52:24.950479 889 tablet_service.cc:1467] Processing CreateTablet for tablet 015412a5fa304453bd8d1be1c3cd7a46 (DEFAULT_TABLE table=default.Table_83a9b08ba3c64e50bbb471dbe4a7ad70 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.950737 888 tablet_service.cc:1467] Processing CreateTablet for tablet e3ee5d8a17874d7ba54de51568e7d2e6 (DEFAULT_TABLE table=default.Table_10e019e7a9244dbeac8bd2a55f618830 [id=2c077da077834ae4bcc84efdd9e89120]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.950927 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 015412a5fa304453bd8d1be1c3cd7a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.951221 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3ee5d8a17874d7ba54de51568e7d2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.952665 1173 tablet_bootstrap.cc:654] T 22750265b522471680835e7f98475e6e P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.954977 1034 tablet_service.cc:1467] Processing CreateTablet for tablet e3ee5d8a17874d7ba54de51568e7d2e6 (DEFAULT_TABLE table=default.Table_10e019e7a9244dbeac8bd2a55f618830 [id=2c077da077834ae4bcc84efdd9e89120]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.955264 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3ee5d8a17874d7ba54de51568e7d2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.960136 759 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0c2fad4f3243238641aae56b305bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.960436 759 tablet_service.cc:1514] Processing DeleteTablet for tablet 41e166b3f8414ababbd4a3f07fb90c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:24.960630 751 tablet_service.cc:1467] Processing CreateTablet for tablet e3ee5d8a17874d7ba54de51568e7d2e6 (DEFAULT_TABLE table=default.Table_10e019e7a9244dbeac8bd2a55f618830 [id=2c077da077834ae4bcc84efdd9e89120]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.960985 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3ee5d8a17874d7ba54de51568e7d2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.961015 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet fb4c004397ee480d897e8b125d248829
I20250112 02:52:24.961717 899 tablet_service.cc:1514] Processing DeleteTablet for tablet fb4c004397ee480d897e8b125d248829 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.963229 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 015412a5fa304453bd8d1be1c3cd7a46 (DEFAULT_TABLE table=default.Table_83a9b08ba3c64e50bbb471dbe4a7ad70 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.963526 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 015412a5fa304453bd8d1be1c3cd7a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.964452 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet fb4c004397ee480d897e8b125d248829 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.965113 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 41e166b3f8414ababbd4a3f07fb90c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.968617 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 41e166b3f8414ababbd4a3f07fb90c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:24.969715 524 master_service.cc:946] ReplaceTablet: received request to replace tablet 3c686fdc8896463c81c9f7c5f53a3da2 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:24.970594 519 master_service.cc:946] ReplaceTablet: received request to replace tablet 28abc02b1b21427eae06560b99bcc078 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:24.970710 759 tablet_service.cc:1467] Processing CreateTablet for tablet 7f4e4f1e451b48dc9cdd780300f6888c (DEFAULT_TABLE table=default.Table_096703f878244b8dac2dcf187fd23fd6 [id=e1779f88b20942eaa2c640ac24946fff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.971011 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f4e4f1e451b48dc9cdd780300f6888c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.975837 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0c2fad4f3243238641aae56b305bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.977298 751 tablet_service.cc:1514] Processing DeleteTablet for tablet fb4c004397ee480d897e8b125d248829 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:24.977487 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet fb4c004397ee480d897e8b125d248829 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb4c004397ee480d897e8b125d248829 already in progress: creating tablet
W20250112 02:52:24.977779 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 41e166b3f8414ababbd4a3f07fb90c4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41e166b3f8414ababbd4a3f07fb90c4c already in progress: creating tablet
W20250112 02:52:24.978118 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 4c0c2fad4f3243238641aae56b305bb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c0c2fad4f3243238641aae56b305bb5 already in progress: creating tablet
I20250112 02:52:24.978107 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 7f4e4f1e451b48dc9cdd780300f6888c (DEFAULT_TABLE table=default.Table_096703f878244b8dac2dcf187fd23fd6 [id=e1779f88b20942eaa2c640ac24946fff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.978412 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f4e4f1e451b48dc9cdd780300f6888c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.979143 896 tablet_service.cc:1467] Processing CreateTablet for tablet 7f4e4f1e451b48dc9cdd780300f6888c (DEFAULT_TABLE table=default.Table_096703f878244b8dac2dcf187fd23fd6 [id=e1779f88b20942eaa2c640ac24946fff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.979326 751 tablet_service.cc:1467] Processing CreateTablet for tablet c04b64bc2f8348a086ed3c4e59f09b16 (DEFAULT_TABLE table=default.Table_2ce39f46be3243acb7f7f99a0e9ac9fb [id=e53d0b4d7340456896f1acdcc50db840]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.979547 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f4e4f1e451b48dc9cdd780300f6888c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.979770 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c04b64bc2f8348a086ed3c4e59f09b16. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:24.983095 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 4c0c2fad4f3243238641aae56b305bb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c0c2fad4f3243238641aae56b305bb5 already in progress: creating tablet
W20250112 02:52:24.983734 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 41e166b3f8414ababbd4a3f07fb90c4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41e166b3f8414ababbd4a3f07fb90c4c already in progress: creating tablet
W20250112 02:52:24.984925 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet fb4c004397ee480d897e8b125d248829 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb4c004397ee480d897e8b125d248829 already in progress: creating tablet
I20250112 02:52:24.985431 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_0b78c0ecb1f245838a0117009873c0e9" } new_table_name: "default.Table_b0af74e9ea8a4c5faea64696a16d5a9a" modify_external_catalogs: true
I20250112 02:52:24.986943 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_34c47be51dd0466ca34b2fdc2bba8919"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:24.987406 1177 tablet_bootstrap.cc:492] T e8220011788a480eb50ee4b670c45739 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
W20250112 02:52:24.987428 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34c47be51dd0466ca34b2fdc2bba8919 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:24.987571 1177 ts_tablet_manager.cc:1397] T e8220011788a480eb50ee4b670c45739 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.006s
I20250112 02:52:24.988360 1177 raft_consensus.cc:357] T e8220011788a480eb50ee4b670c45739 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.988559 1177 raft_consensus.cc:383] T e8220011788a480eb50ee4b670c45739 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:24.988643 1177 raft_consensus.cc:738] T e8220011788a480eb50ee4b670c45739 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:24.988936 1177 consensus_queue.cc:260] T e8220011788a480eb50ee4b670c45739 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:24.989073 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0030637279449fa9c00cdc075fbca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:24.989799 525 catalog_manager.cc:5526] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:24.990212 525 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5a8e1b96accd459eaed221ded4a453b6 (table default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94]). Expected version 2 got 0
I20250112 02:52:24.990305 888 tablet_service.cc:1467] Processing CreateTablet for tablet c04b64bc2f8348a086ed3c4e59f09b16 (DEFAULT_TABLE table=default.Table_2ce39f46be3243acb7f7f99a0e9ac9fb [id=e53d0b4d7340456896f1acdcc50db840]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.990494 519 master_service.cc:946] ReplaceTablet: received request to replace tablet b885247d2cb449e0a29f6f5d81518589 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:24.990625 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c04b64bc2f8348a086ed3c4e59f09b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.991183 1177 ts_tablet_manager.cc:1428] T e8220011788a480eb50ee4b670c45739 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:24.991636 1172 tablet_replica.cc:331] T bb0030637279449fa9c00cdc075fbca9 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:24.991775 1177 tablet_bootstrap.cc:492] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:24.991802 1172 raft_consensus.cc:2238] T bb0030637279449fa9c00cdc075fbca9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:24.991951 1172 raft_consensus.cc:2267] T bb0030637279449fa9c00cdc075fbca9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:24.993541 1172 ts_tablet_manager.cc:1905] T bb0030637279449fa9c00cdc075fbca9 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:24.993638 754 tablet_service.cc:1467] Processing CreateTablet for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 (DEFAULT_TABLE table=default.Table_f4d612cd6a524bc493147493c5a4a284 [id=4ac16bc62bbe4ed7a977d22c92ad580d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:24.993889 1177 tablet_bootstrap.cc:654] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:24.993954 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6913b0ac9ceb4dbc9cdf591f159134f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:24.999517 759 tablet_service.cc:1514] Processing DeleteTablet for tablet e8220011788a480eb50ee4b670c45739 with delete_type TABLET_DATA_DELETED (replaced by tablet b885247d2cb449e0a29f6f5d81518589) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.002094 1172 ts_tablet_manager.cc:1918] T bb0030637279449fa9c00cdc075fbca9 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:25.002499 1172 log.cc:1198] T bb0030637279449fa9c00cdc075fbca9 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/bb0030637279449fa9c00cdc075fbca9
I20250112 02:52:25.002936 1172 ts_tablet_manager.cc:1939] T bb0030637279449fa9c00cdc075fbca9 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:25.003027 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_e3301643d2294e8ea5f9c3c1bc366eb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:25.003398 1034 tablet_service.cc:1467] Processing CreateTablet for tablet c04b64bc2f8348a086ed3c4e59f09b16 (DEFAULT_TABLE table=default.Table_2ce39f46be3243acb7f7f99a0e9ac9fb [id=e53d0b4d7340456896f1acdcc50db840]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.003444 1175 ts_tablet_manager.cc:1905] T aa67bf25dc894d7ea96d247a22c14e03 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:25.003535 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3301643d2294e8ea5f9c3c1bc366eb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.003890 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c04b64bc2f8348a086ed3c4e59f09b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.007622 1171 tablet_replica.cc:331] T e8220011788a480eb50ee4b670c45739 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:25.007876 1171 raft_consensus.cc:2238] T e8220011788a480eb50ee4b670c45739 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:25.008062 1171 raft_consensus.cc:2267] T e8220011788a480eb50ee4b670c45739 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:25.008895 1171 ts_tablet_manager.cc:1905] T e8220011788a480eb50ee4b670c45739 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:25.012977 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 (DEFAULT_TABLE table=default.Table_f4d612cd6a524bc493147493c5a4a284 [id=4ac16bc62bbe4ed7a977d22c92ad580d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.013273 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6913b0ac9ceb4dbc9cdf591f159134f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.014420 1173 tablet_bootstrap.cc:492] T 22750265b522471680835e7f98475e6e P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:25.014619 1173 ts_tablet_manager.cc:1397] T 22750265b522471680835e7f98475e6e P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.073s user 0.000s sys 0.009s
I20250112 02:52:25.015386 1173 raft_consensus.cc:357] T 22750265b522471680835e7f98475e6e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.015662 1173 raft_consensus.cc:383] T 22750265b522471680835e7f98475e6e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.015771 1173 raft_consensus.cc:738] T 22750265b522471680835e7f98475e6e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.016041 1173 consensus_queue.cc:260] T 22750265b522471680835e7f98475e6e P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.016959 1173 ts_tablet_manager.cc:1428] T 22750265b522471680835e7f98475e6e P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:25.017270 1173 tablet_bootstrap.cc:492] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:25.017760 512 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 54033a747d5a49b0a1c16f138158f596 (table default.Table_4f66980b946046a4bd2531c4ab094352 [id=c11cab752ad14d53b9b09fbaaf9b6f36], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 54033a747d5a49b0a1c16f138158f596 consensus state has no leader
I20250112 02:52:25.019649 1173 tablet_bootstrap.cc:654] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.020421 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 97b31ebe035c4552baaffc3133ec73c1
I20250112 02:52:25.020479 896 tablet_service.cc:1467] Processing CreateTablet for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 (DEFAULT_TABLE table=default.Table_f4d612cd6a524bc493147493c5a4a284 [id=4ac16bc62bbe4ed7a977d22c92ad580d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.020884 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6913b0ac9ceb4dbc9cdf591f159134f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.022094 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_3a84890c771c4574a25d33543fe7217d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.023003 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a84890c771c4574a25d33543fe7217d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.023504 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 97b31ebe035c4552baaffc3133ec73c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.024109 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 97b31ebe035c4552baaffc3133ec73c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.026075 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 97b31ebe035c4552baaffc3133ec73c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.026428 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet bb0030637279449fa9c00cdc075fbca9 (table default.Table_12a6d3a2f70d4920b28aa77c6654b93e [id=c77dc91fbb544fb49d244b044ab51c26]) successfully deleted
W20250112 02:52:25.027092 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 97b31ebe035c4552baaffc3133ec73c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97b31ebe035c4552baaffc3133ec73c1 already in progress: creating tablet
I20250112 02:52:25.028379 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet afa2f8aabe5e42ccbc15b193fdc03563
I20250112 02:52:25.028965 522 master_service.cc:946] ReplaceTablet: received request to replace tablet 97b31ebe035c4552baaffc3133ec73c1 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:25.029510 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_a6b320592c334f8bb0ff1a1ddf3ab0c0" } new_table_name: "default.Table_498525c117bf4424a217c15eda6ba3ce" modify_external_catalogs: true
I20250112 02:52:25.030521 1175 ts_tablet_manager.cc:1918] T aa67bf25dc894d7ea96d247a22c14e03 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:25.030654 1175 log.cc:1198] T aa67bf25dc894d7ea96d247a22c14e03 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/aa67bf25dc894d7ea96d247a22c14e03
I20250112 02:52:25.030905 896 tablet_service.cc:1514] Processing DeleteTablet for tablet afa2f8aabe5e42ccbc15b193fdc03563 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.030984 1175 ts_tablet_manager.cc:1939] T aa67bf25dc894d7ea96d247a22c14e03 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:25.031625 751 tablet_service.cc:1514] Processing DeleteTablet for tablet afa2f8aabe5e42ccbc15b193fdc03563 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.033486 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet afa2f8aabe5e42ccbc15b193fdc03563 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
W20250112 02:52:25.033951 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet afa2f8aabe5e42ccbc15b193fdc03563 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afa2f8aabe5e42ccbc15b193fdc03563 already in progress: creating tablet
I20250112 02:52:25.034232 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 22750265b522471680835e7f98475e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.034463 1172 tablet_replica.cc:331] T 22750265b522471680835e7f98475e6e P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:25.034613 1172 raft_consensus.cc:2238] T 22750265b522471680835e7f98475e6e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:25.034778 1172 raft_consensus.cc:2267] T 22750265b522471680835e7f98475e6e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:25.035108 1171 ts_tablet_manager.cc:1918] T e8220011788a480eb50ee4b670c45739 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:25.035243 1171 log.cc:1198] T e8220011788a480eb50ee4b670c45739 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/e8220011788a480eb50ee4b670c45739
I20250112 02:52:25.036154 1172 ts_tablet_manager.cc:1905] T 22750265b522471680835e7f98475e6e P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:25.039314 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet aa67bf25dc894d7ea96d247a22c14e03 (table default.Table_7e88ecc0aa5e43ffb84af9b8dc6da938 [id=2f9194b28f5044e2a85f9e18fa041746]) successfully deleted
W20250112 02:52:25.039546 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet b7bd96fb381f4e73a0081be3fc799c36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7bd96fb381f4e73a0081be3fc799c36 already in progress: creating tablet
W20250112 02:52:25.039774 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 already in progress: creating tablet
I20250112 02:52:25.039935 1190 raft_consensus.cc:491] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 02:52:25.039968 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 4c0c2fad4f3243238641aae56b305bb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c0c2fad4f3243238641aae56b305bb5 already in progress: creating tablet
W20250112 02:52:25.040194 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet fb4c004397ee480d897e8b125d248829 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb4c004397ee480d897e8b125d248829 already in progress: creating tablet
I20250112 02:52:25.040129 1190 raft_consensus.cc:513] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.040930 1190 leader_election.cc:290] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:25.041297 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_f359341b73c14c60b859f06dc5f428a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:25.041472 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2595782d66e4e4b890c94221b196daa" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:25.041659 780 raft_consensus.cc:2463] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:25.041759 1172 ts_tablet_manager.cc:1918] T 22750265b522471680835e7f98475e6e P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 02:52:25.041795 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f359341b73c14c60b859f06dc5f428a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.041880 1172 log.cc:1198] T 22750265b522471680835e7f98475e6e P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/22750265b522471680835e7f98475e6e
I20250112 02:52:25.042552 855 leader_election.cc:304] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:25.042624 1171 ts_tablet_manager.cc:1939] T e8220011788a480eb50ee4b670c45739 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:25.042551 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2595782d66e4e4b890c94221b196daa" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:25.042208 1172 ts_tablet_manager.cc:1939] T 22750265b522471680835e7f98475e6e P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:25.042923 1190 raft_consensus.cc:2798] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:25.043000 1190 raft_consensus.cc:491] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:25.043083 1190 raft_consensus.cc:3054] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:25.043540 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet e8220011788a480eb50ee4b670c45739 (table default.Table_edaa8b50dfa8463e99e05ab8d1d89bc3 [id=a6a18be9a78e4959b7d272c447f6fc25]) successfully deleted
W20250112 02:52:25.044394 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 41e166b3f8414ababbd4a3f07fb90c4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41e166b3f8414ababbd4a3f07fb90c4c already in progress: creating tablet
W20250112 02:52:25.045022 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 97b31ebe035c4552baaffc3133ec73c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97b31ebe035c4552baaffc3133ec73c1 already in progress: creating tablet
W20250112 02:52:25.045727 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet afa2f8aabe5e42ccbc15b193fdc03563 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afa2f8aabe5e42ccbc15b193fdc03563 already in progress: creating tablet
W20250112 02:52:25.045725 854 leader_election.cc:343] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:25.046682 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 97b31ebe035c4552baaffc3133ec73c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97b31ebe035c4552baaffc3133ec73c1 already in progress: creating tablet
I20250112 02:52:25.046991 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 22750265b522471680835e7f98475e6e (table default.Table_89761f48fc8547e8b786c9559485f243 [id=c8799d47318d42d9859e36a37a5a9445]) successfully deleted
I20250112 02:52:25.046983 1190 raft_consensus.cc:513] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:25.047976 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet afa2f8aabe5e42ccbc15b193fdc03563 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afa2f8aabe5e42ccbc15b193fdc03563 already in progress: creating tablet
I20250112 02:52:25.048002 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2595782d66e4e4b890c94221b196daa" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:25.048157 780 raft_consensus.cc:3054] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:25.049470 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c04b64bc2f8348a086ed3c4e59f09b16 consensus state has no leader
I20250112 02:52:25.050009 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_f89a1b732e34437294f2a7cdc052b6ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.050602 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f89a1b732e34437294f2a7cdc052b6ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.051973 780 raft_consensus.cc:2463] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:25.051903 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_9f27fd5786f84a8084cdf01f95f4519b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.052497 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f27fd5786f84a8084cdf01f95f4519b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.052603 855 leader_election.cc:304] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:25.052961 1269 raft_consensus.cc:2798] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:25.053130 1269 raft_consensus.cc:695] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:25.053465 1269 consensus_queue.cc:237] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.054381 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_3c1f0242d17443b5a758967cf82ca0ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.054816 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c1f0242d17443b5a758967cf82ca0ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.056706 1190 leader_election.cc:290] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:25.056950 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2595782d66e4e4b890c94221b196daa" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:25.059348 1259 tablet.cc:1721] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:25.060123 1260 tablet.cc:1721] T 3f5b418c17084420b867498fda79fb0a P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 02:52:25.060627 854 leader_election.cc:343] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.072961 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 3c686fdc8896463c81c9f7c5f53a3da2
I20250112 02:52:25.073493 524 catalog_manager.cc:6596] ReplaceTablet: tablet 3c686fdc8896463c81c9f7c5f53a3da2 deleted and replaced by tablet 2c03b5e612d64314b210c11686fdfa2a
I20250112 02:52:25.076014 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c686fdc8896463c81c9f7c5f53a3da2 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c03b5e612d64314b210c11686fdfa2a) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.076202 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_830855385a78429f908678179b435e11" } new_table_name: "default.Table_3b1c87a5169e49b68fdf3d5c662a27bc" modify_external_catalogs: true
W20250112 02:52:25.076779 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 3c686fdc8896463c81c9f7c5f53a3da2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c686fdc8896463c81c9f7c5f53a3da2 already in progress: creating tablet
I20250112 02:52:25.079193 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c686fdc8896463c81c9f7c5f53a3da2 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c03b5e612d64314b210c11686fdfa2a) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.079818 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c686fdc8896463c81c9f7c5f53a3da2 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c03b5e612d64314b210c11686fdfa2a) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:25.080652 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 3c686fdc8896463c81c9f7c5f53a3da2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c686fdc8896463c81c9f7c5f53a3da2 already in progress: creating tablet
W20250112 02:52:25.085072 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 3c686fdc8896463c81c9f7c5f53a3da2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c686fdc8896463c81c9f7c5f53a3da2 already in progress: creating tablet
I20250112 02:52:25.093345 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 09886cd08c714f24a96e4df02bc0e6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.093679 1175 tablet_replica.cc:331] T 09886cd08c714f24a96e4df02bc0e6c8 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:25.093930 1175 raft_consensus.cc:2238] T 09886cd08c714f24a96e4df02bc0e6c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:25.094147 1175 raft_consensus.cc:2267] T 09886cd08c714f24a96e4df02bc0e6c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:25.094930 1175 ts_tablet_manager.cc:1905] T 09886cd08c714f24a96e4df02bc0e6c8 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:25.096624 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_3888b008030c447d885c3961d470cee2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.097225 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3888b008030c447d885c3961d470cee2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.103212 1175 ts_tablet_manager.cc:1918] T 09886cd08c714f24a96e4df02bc0e6c8 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:25.103410 1175 log.cc:1198] T 09886cd08c714f24a96e4df02bc0e6c8 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/09886cd08c714f24a96e4df02bc0e6c8
I20250112 02:52:25.103909 1175 ts_tablet_manager.cc:1939] T 09886cd08c714f24a96e4df02bc0e6c8 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:25.105113 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 09886cd08c714f24a96e4df02bc0e6c8 (table default.Table_f82a967adb9040f69df9732d0d228d8d [id=fce07a88937f4f5d8b14483e056bcb4d]) successfully deleted
I20250112 02:52:25.107301 1181 tablet_bootstrap.cc:492] T f05d3326545849b9af4519bc66c21069 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:25.107563 1181 ts_tablet_manager.cc:1397] T f05d3326545849b9af4519bc66c21069 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.191s user 0.029s sys 0.027s
I20250112 02:52:25.108574 1181 raft_consensus.cc:357] T f05d3326545849b9af4519bc66c21069 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.108934 1181 raft_consensus.cc:383] T f05d3326545849b9af4519bc66c21069 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.109072 1181 raft_consensus.cc:738] T f05d3326545849b9af4519bc66c21069 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.109433 1181 consensus_queue.cc:260] T f05d3326545849b9af4519bc66c21069 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.111006 1181 ts_tablet_manager.cc:1428] T f05d3326545849b9af4519bc66c21069 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:25.111428 1181 tablet_bootstrap.cc:492] T 28abc02b1b21427eae06560b99bcc078 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:25.113871 1181 tablet_bootstrap.cc:654] T 28abc02b1b21427eae06560b99bcc078 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.129002 1173 tablet_bootstrap.cc:492] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:25.129153 1173 ts_tablet_manager.cc:1397] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.112s user 0.027s sys 0.041s
W20250112 02:52:25.129567 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 65f22a7b1b804661af3b7e785a833739 consensus state has no leader
I20250112 02:52:25.129920 1173 raft_consensus.cc:357] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.130126 1173 raft_consensus.cc:383] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.130211 1173 raft_consensus.cc:738] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.130474 1173 consensus_queue.cc:260] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.130882 1177 tablet_bootstrap.cc:492] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:25.131006 1177 ts_tablet_manager.cc:1397] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.139s user 0.006s sys 0.025s
I20250112 02:52:25.131870 1177 raft_consensus.cc:357] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.132097 1177 raft_consensus.cc:383] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.132031 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_0f7c10fd7b6245f7a7652d516708fe08" } new_table_name: "default.Table_3fcca85a33054d7ab388f29bba85fe35" modify_external_catalogs: true
I20250112 02:52:25.132553 1177 raft_consensus.cc:738] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.133364 1173 ts_tablet_manager.cc:1428] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 02:52:25.132927 1177 consensus_queue.cc:260] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.134876 1177 ts_tablet_manager.cc:1428] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250112 02:52:25.135409 1173 tablet_bootstrap.cc:492] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:25.136831 1177 tablet_bootstrap.cc:492] T 09886cd08c714f24a96e4df02bc0e6c8 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:25.136885 1173 tablet_bootstrap.cc:654] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.139716 1181 tablet_bootstrap.cc:492] T 28abc02b1b21427eae06560b99bcc078 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:25.139914 1181 ts_tablet_manager.cc:1397] T 28abc02b1b21427eae06560b99bcc078 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.022s
I20250112 02:52:25.140604 1177 tablet_bootstrap.cc:654] T 09886cd08c714f24a96e4df02bc0e6c8 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.140776 1181 raft_consensus.cc:357] T 28abc02b1b21427eae06560b99bcc078 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.141064 1181 raft_consensus.cc:383] T 28abc02b1b21427eae06560b99bcc078 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.141247 1181 raft_consensus.cc:738] T 28abc02b1b21427eae06560b99bcc078 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.141551 1178 raft_consensus.cc:491] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:25.141535 1181 consensus_queue.cc:260] T 28abc02b1b21427eae06560b99bcc078 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.141717 1178 raft_consensus.cc:513] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.142318 1178 leader_election.cc:290] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:25.142827 1181 ts_tablet_manager.cc:1428] T 28abc02b1b21427eae06560b99bcc078 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 02:52:25.142952 518 master_service.cc:946] ReplaceTablet: received request to replace tablet b885247d2cb449e0a29f6f5d81518589 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:25.143124 1181 tablet_bootstrap.cc:492] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
W20250112 02:52:25.143779 991 leader_election.cc:343] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: 2b34fb96bc5140118725af8346ea827f
I20250112 02:52:25.143997 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b34fb96bc5140118725af8346ea827f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.144327 1172 tablet_replica.cc:331] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:25.144475 1172 raft_consensus.cc:2238] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:25.144637 1172 raft_consensus.cc:2267] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:25.145452 1181 tablet_bootstrap.cc:654] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:25.145789 989 leader_election.cc:343] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 2b34fb96bc5140118725af8346ea827f
I20250112 02:52:25.145896 989 leader_election.cc:304] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
W20250112 02:52:25.146067 989 raft_consensus.cc:2681] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 02:52:25.147017 1172 ts_tablet_manager.cc:1905] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:25.152987 1172 ts_tablet_manager.cc:1918] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:25.153002 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_d38ef91002ba4653916c476d1df3a9c6" } modify_external_catalogs: true
I20250112 02:52:25.153180 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_d38ef91002ba4653916c476d1df3a9c6" } modify_external_catalogs: true
I20250112 02:52:25.153903 1181 tablet_bootstrap.cc:492] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:25.154042 1181 ts_tablet_manager.cc:1397] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20250112 02:52:25.154167 1177 tablet_bootstrap.cc:492] T 09886cd08c714f24a96e4df02bc0e6c8 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:25.153587 1173 tablet_bootstrap.cc:492] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:25.154294 1177 ts_tablet_manager.cc:1397] T 09886cd08c714f24a96e4df02bc0e6c8 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.002s
I20250112 02:52:25.154433 1172 log.cc:1198] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/2b34fb96bc5140118725af8346ea827f
I20250112 02:52:25.154737 1181 raft_consensus.cc:357] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.154575 1173 ts_tablet_manager.cc:1397] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.006s
I20250112 02:52:25.154932 1181 raft_consensus.cc:383] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.155025 1181 raft_consensus.cc:738] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.154969 1177 raft_consensus.cc:357] T 09886cd08c714f24a96e4df02bc0e6c8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.155175 1177 raft_consensus.cc:383] T 09886cd08c714f24a96e4df02bc0e6c8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.155249 1177 raft_consensus.cc:738] T 09886cd08c714f24a96e4df02bc0e6c8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.155272 1181 consensus_queue.cc:260] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.155519 1177 consensus_queue.cc:260] T 09886cd08c714f24a96e4df02bc0e6c8 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.156062 1177 ts_tablet_manager.cc:1428] T 09886cd08c714f24a96e4df02bc0e6c8 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:25.155979 1173 raft_consensus.cc:357] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.156333 1177 tablet_bootstrap.cc:492] T 28abc02b1b21427eae06560b99bcc078 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:25.156219 1172 ts_tablet_manager.cc:1939] T 2b34fb96bc5140118725af8346ea827f P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:25.156363 1173 raft_consensus.cc:383] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.157032 1173 raft_consensus.cc:738] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.157688 1173 consensus_queue.cc:260] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.157900 1181 ts_tablet_manager.cc:1428] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250112 02:52:25.158701 1181 tablet_bootstrap.cc:492] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:25.158989 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 2b34fb96bc5140118725af8346ea827f (table default.Table_c2d18d092b5a46f8b59f64aa4a8f02c7 [id=059958c4e1c74d8d8e4c029030d85895]) successfully deleted
I20250112 02:52:25.161082 1181 tablet_bootstrap.cc:654] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.162019 1173 ts_tablet_manager.cc:1428] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20250112 02:52:25.163327 1173 tablet_bootstrap.cc:492] T 2728a022ef0d4952966d397a32356503 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:25.164506 1177 tablet_bootstrap.cc:654] T 28abc02b1b21427eae06560b99bcc078 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.165493 1173 tablet_bootstrap.cc:654] T 2728a022ef0d4952966d397a32356503 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.167995 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_426c5d1d19a94f0d8110a9cbee9a9044" } modify_external_catalogs: true
I20250112 02:52:25.168232 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_426c5d1d19a94f0d8110a9cbee9a9044" } modify_external_catalogs: true
I20250112 02:52:25.169246 526 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_945067c243f84394957a328d65592441" } modify_external_catalogs: true
I20250112 02:52:25.169472 526 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_945067c243f84394957a328d65592441" } modify_external_catalogs: true
I20250112 02:52:25.169793 1181 tablet_bootstrap.cc:492] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:25.169984 1181 ts_tablet_manager.cc:1397] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20250112 02:52:25.170843 1181 raft_consensus.cc:357] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.171048 1181 raft_consensus.cc:383] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.171152 1181 raft_consensus.cc:738] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.171419 1181 consensus_queue.cc:260] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.172542 1177 tablet_bootstrap.cc:492] T 28abc02b1b21427eae06560b99bcc078 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:25.172680 1177 ts_tablet_manager.cc:1397] T 28abc02b1b21427eae06560b99bcc078 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.006s
I20250112 02:52:25.173573 1181 ts_tablet_manager.cc:1428] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:25.174144 1177 raft_consensus.cc:357] T 28abc02b1b21427eae06560b99bcc078 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.174324 1177 raft_consensus.cc:383] T 28abc02b1b21427eae06560b99bcc078 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.174404 1177 raft_consensus.cc:738] T 28abc02b1b21427eae06560b99bcc078 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.174547 1173 tablet_bootstrap.cc:492] T 2728a022ef0d4952966d397a32356503 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:25.174729 1173 ts_tablet_manager.cc:1397] T 2728a022ef0d4952966d397a32356503 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20250112 02:52:25.174638 1177 consensus_queue.cc:260] T 28abc02b1b21427eae06560b99bcc078 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.175660 1177 ts_tablet_manager.cc:1428] T 28abc02b1b21427eae06560b99bcc078 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 02:52:25.175908 1173 raft_consensus.cc:357] T 2728a022ef0d4952966d397a32356503 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.176126 1173 raft_consensus.cc:383] T 2728a022ef0d4952966d397a32356503 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.176258 1173 raft_consensus.cc:738] T 2728a022ef0d4952966d397a32356503 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.176729 1173 consensus_queue.cc:260] T 2728a022ef0d4952966d397a32356503 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.177263 1177 tablet_bootstrap.cc:492] T 235e9dd0964c4ea6af7d05ef58acae0d P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:25.178200 1173 ts_tablet_manager.cc:1428] T 2728a022ef0d4952966d397a32356503 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 02:52:25.178687 1173 tablet_bootstrap.cc:492] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:25.178925 1181 tablet_bootstrap.cc:492] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:25.180544 1173 tablet_bootstrap.cc:654] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.180714 1181 tablet_bootstrap.cc:654] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.182412 1177 tablet_bootstrap.cc:654] T 235e9dd0964c4ea6af7d05ef58acae0d P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.187433 1177 tablet_bootstrap.cc:492] T 235e9dd0964c4ea6af7d05ef58acae0d P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:25.187639 1177 ts_tablet_manager.cc:1397] T 235e9dd0964c4ea6af7d05ef58acae0d P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20250112 02:52:25.189241 1177 raft_consensus.cc:357] T 235e9dd0964c4ea6af7d05ef58acae0d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.189450 1173 tablet_bootstrap.cc:492] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:25.189512 1177 raft_consensus.cc:383] T 235e9dd0964c4ea6af7d05ef58acae0d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.189589 1177 raft_consensus.cc:738] T 235e9dd0964c4ea6af7d05ef58acae0d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.189790 1173 ts_tablet_manager.cc:1397] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20250112 02:52:25.189909 1177 consensus_queue.cc:260] T 235e9dd0964c4ea6af7d05ef58acae0d P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.190660 1173 raft_consensus.cc:357] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.190898 1173 raft_consensus.cc:383] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.191036 1173 raft_consensus.cc:738] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.191320 1173 consensus_queue.cc:260] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.192502 1193 raft_consensus.cc:491] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:25.192672 1193 raft_consensus.cc:513] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.193029 1173 ts_tablet_manager.cc:1428] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:25.193261 1173 tablet_bootstrap.cc:492] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:25.193396 1193 leader_election.cc:290] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:25.194432 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05d3326545849b9af4519bc66c21069" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:25.194746 919 raft_consensus.cc:2463] T f05d3326545849b9af4519bc66c21069 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:25.196106 713 leader_election.cc:304] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
W20250112 02:52:25.198109 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5601a9398c9a466582a98967c0e749e2 consensus state has no leader
I20250112 02:52:25.198956 1193 raft_consensus.cc:2798] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:25.199096 1193 raft_consensus.cc:491] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:25.199193 1193 raft_consensus.cc:3054] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:25.199640 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05d3326545849b9af4519bc66c21069" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:25.200407 714 leader_election.cc:343] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.202848 1173 tablet_bootstrap.cc:654] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.203729 1193 raft_consensus.cc:513] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.204674 521 master_service.cc:946] ReplaceTablet: received request to replace tablet 3567820f0d4349799b681fdfff0ebad8 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:25.204891 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05d3326545849b9af4519bc66c21069" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:25.205134 919 raft_consensus.cc:3054] T f05d3326545849b9af4519bc66c21069 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:25.208565 919 raft_consensus.cc:2463] T f05d3326545849b9af4519bc66c21069 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:25.209139 713 leader_election.cc:304] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:25.209478 1245 raft_consensus.cc:2798] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:25.211089 1173 tablet_bootstrap.cc:492] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:25.211215 1173 ts_tablet_manager.cc:1397] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.007s
I20250112 02:52:25.213186 1193 leader_election.cc:290] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:25.214007 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05d3326545849b9af4519bc66c21069" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:25.214514 714 leader_election.cc:343] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.211954 1173 raft_consensus.cc:357] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.214712 1173 raft_consensus.cc:383] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.214834 1173 raft_consensus.cc:738] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.215108 1173 consensus_queue.cc:260] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.216254 1173 ts_tablet_manager.cc:1428] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 02:52:25.216826 520 master_service.cc:946] ReplaceTablet: received request to replace tablet 3fd498ccce484a89855c982762086890 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:25.217170 1173 tablet_bootstrap.cc:492] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:25.218708 1173 tablet_bootstrap.cc:654] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.219244 1177 ts_tablet_manager.cc:1428] T 235e9dd0964c4ea6af7d05ef58acae0d P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.032s user 0.003s sys 0.000s
I20250112 02:52:25.220517 896 tablet_service.cc:1467] Processing CreateTablet for tablet abfaad67d5e34c0a8338dccc0ef05466 (DEFAULT_TABLE table=default.Table_7d54fa05bdbf4ff3b5cce201222e62de [id=2c0e94e8c5774da1b8130c7651d94500]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.220752 1177 tablet_bootstrap.cc:492] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:25.220887 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abfaad67d5e34c0a8338dccc0ef05466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.221146 519 catalog_manager.cc:5526] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:25.221664 519 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c2595782d66e4e4b890c94221b196daa (table default.Table_96a35d4e94284a49ba1a88d7d0acd3c1 [id=8559c454b17b4a00a60689a6a9b19f7d]). Expected version 1 got 0
I20250112 02:52:25.222322 524 catalog_manager.cc:6071] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: default.Table_b0af74e9ea8a4c5faea64696a16d5a9a [id=ad0be37e6b6f4b9597dcb4cbd50d5b12] alter complete (version 2)
I20250112 02:52:25.222103 1177 tablet_bootstrap.cc:654] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.224079 754 tablet_service.cc:1467] Processing CreateTablet for tablet abfaad67d5e34c0a8338dccc0ef05466 (DEFAULT_TABLE table=default.Table_7d54fa05bdbf4ff3b5cce201222e62de [id=2c0e94e8c5774da1b8130c7651d94500]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.224490 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abfaad67d5e34c0a8338dccc0ef05466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.233897 1020 tablet_service.cc:1467] Processing CreateTablet for tablet abfaad67d5e34c0a8338dccc0ef05466 (DEFAULT_TABLE table=default.Table_7d54fa05bdbf4ff3b5cce201222e62de [id=2c0e94e8c5774da1b8130c7651d94500]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.234238 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abfaad67d5e34c0a8338dccc0ef05466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.238536 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_981a70d4d3184fd8a99e6b81ae1caab7" } modify_external_catalogs: true
I20250112 02:52:25.238706 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_981a70d4d3184fd8a99e6b81ae1caab7" } modify_external_catalogs: true
I20250112 02:52:25.239567 1245 raft_consensus.cc:695] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:25.239884 1245 consensus_queue.cc:237] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.240329 1181 tablet_bootstrap.cc:492] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:25.240505 1181 ts_tablet_manager.cc:1397] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.062s user 0.012s sys 0.027s
I20250112 02:52:25.240676 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_fd0169db0f984d3cbc0d8624dda686bc" } modify_external_catalogs: true
I20250112 02:52:25.240875 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_fd0169db0f984d3cbc0d8624dda686bc" } modify_external_catalogs: true
I20250112 02:52:25.241382 1181 raft_consensus.cc:357] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.241634 1181 raft_consensus.cc:383] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.241703 1181 raft_consensus.cc:738] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.241992 1181 consensus_queue.cc:260] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.245398 1181 ts_tablet_manager.cc:1428] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 02:52:25.247967 1181 tablet_bootstrap.cc:492] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:25.249382 1181 tablet_bootstrap.cc:654] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:25.252633 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3567820f0d4349799b681fdfff0ebad8 consensus state has no leader
I20250112 02:52:25.253742 1181 tablet_bootstrap.cc:492] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:25.253870 1181 ts_tablet_manager.cc:1397] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20250112 02:52:25.254487 1181 raft_consensus.cc:357] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.254683 1181 raft_consensus.cc:383] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.254774 1181 raft_consensus.cc:738] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.254964 1181 consensus_queue.cc:260] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.256181 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_3bc5157a35854b91b0c087c3ea919965"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.256731 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bc5157a35854b91b0c087c3ea919965 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.257918 1181 ts_tablet_manager.cc:1428] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 02:52:25.258136 1177 tablet_bootstrap.cc:492] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:25.258275 1177 ts_tablet_manager.cc:1397] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.002s
I20250112 02:52:25.259021 1177 raft_consensus.cc:357] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.259225 1177 raft_consensus.cc:383] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.259346 1177 raft_consensus.cc:738] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.259635 1181 tablet_bootstrap.cc:492] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:25.260133 1178 raft_consensus.cc:491] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:25.260308 1178 raft_consensus.cc:513] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.260371 1177 consensus_queue.cc:260] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.261790 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2595782d66e4e4b890c94221b196daa" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:25.262049 780 raft_consensus.cc:2388] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1e775296eb746db8b318f17d26820e6 in current term 1: Already voted for candidate fe36951a83fb4e57b26e187edbcbfe3d in this term.
I20250112 02:52:25.262503 1181 tablet_bootstrap.cc:654] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.262825 1178 leader_election.cc:290] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:25.262919 1177 ts_tablet_manager.cc:1428] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 02:52:25.263389 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2595782d66e4e4b890c94221b196daa" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:25.266155 1177 tablet_bootstrap.cc:492] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:25.268196 1177 tablet_bootstrap.cc:654] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.270385 1181 tablet_bootstrap.cc:492] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:25.270516 1181 ts_tablet_manager.cc:1397] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20250112 02:52:25.271103 1181 raft_consensus.cc:357] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.271282 1181 raft_consensus.cc:383] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.271287 526 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 3fd498ccce484a89855c982762086890
I20250112 02:52:25.271338 1181 raft_consensus.cc:738] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.271633 1181 consensus_queue.cc:260] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.272984 1181 ts_tablet_manager.cc:1428] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:25.273401 1181 tablet_bootstrap.cc:492] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:25.273620 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd498ccce484a89855c982762086890 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.274924 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd498ccce484a89855c982762086890 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:25.275749 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 3fd498ccce484a89855c982762086890 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fd498ccce484a89855c982762086890 already in progress: creating tablet
W20250112 02:52:25.275761 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 3fd498ccce484a89855c982762086890 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fd498ccce484a89855c982762086890 already in progress: creating tablet
I20250112 02:52:25.277805 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_17d23578eb794e99829a550bed4a11d9" } new_table_name: "default.Table_d1e5cef5fa3e4d90bada176f80bd4180" modify_external_catalogs: true
I20250112 02:52:25.278424 1177 tablet_bootstrap.cc:492] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:25.278548 1177 ts_tablet_manager.cc:1397] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20250112 02:52:25.278995 1181 tablet_bootstrap.cc:654] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.279268 1177 raft_consensus.cc:357] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.279500 1177 raft_consensus.cc:383] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.279582 1177 raft_consensus.cc:738] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.279886 1177 consensus_queue.cc:260] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.283504 1177 ts_tablet_manager.cc:1428] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250112 02:52:25.283885 989 leader_election.cc:304] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:25.284821 1178 raft_consensus.cc:491] T 3f5b418c17084420b867498fda79fb0a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:25.284739 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd498ccce484a89855c982762086890 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.285379 1178 raft_consensus.cc:513] T 3f5b418c17084420b867498fda79fb0a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.286849 1177 tablet_bootstrap.cc:492] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:25.287356 1181 tablet_bootstrap.cc:492] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:25.287515 1181 ts_tablet_manager.cc:1397] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
W20250112 02:52:25.287648 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 3fd498ccce484a89855c982762086890 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fd498ccce484a89855c982762086890 already in progress: creating tablet
I20250112 02:52:25.288031 1178 leader_election.cc:290] T 3f5b418c17084420b867498fda79fb0a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:25.288268 1178 raft_consensus.cc:3054] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:25.288327 1181 raft_consensus.cc:357] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.288501 1181 raft_consensus.cc:383] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.288589 1181 raft_consensus.cc:738] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.288837 1181 consensus_queue.cc:260] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.289011 1177 tablet_bootstrap.cc:654] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.291172 1181 ts_tablet_manager.cc:1428] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:25.292091 1177 tablet_bootstrap.cc:492] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:25.292047 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5b418c17084420b867498fda79fb0a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:25.292224 1177 ts_tablet_manager.cc:1397] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20250112 02:52:25.292891 1177 raft_consensus.cc:357] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.293290 1181 tablet_bootstrap.cc:492] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:25.293399 1178 raft_consensus.cc:2743] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:25.293089 1177 raft_consensus.cc:383] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.293879 1177 raft_consensus.cc:738] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.294103 1177 consensus_queue.cc:260] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.295116 1269 raft_consensus.cc:491] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:25.295290 1269 raft_consensus.cc:513] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.296038 1269 leader_election.cc:290] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:25.296521 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6ccd04880442d0990c6cb0d27956bd" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:25.296942 855 leader_election.cc:343] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.297205 1177 ts_tablet_manager.cc:1428] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 02:52:25.297317 1181 tablet_bootstrap.cc:654] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.300642 1177 tablet_bootstrap.cc:492] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:25.300907 1269 raft_consensus.cc:491] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:25.301024 1269 raft_consensus.cc:513] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.301563 1269 leader_election.cc:290] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:25.302121 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5b418c17084420b867498fda79fb0a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:25.302110 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6ccd04880442d0990c6cb0d27956bd" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:25.302218 1177 tablet_bootstrap.cc:654] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.302623 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5601a9398c9a466582a98967c0e749e2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:25.302659 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5601a9398c9a466582a98967c0e749e2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:25.302856 780 raft_consensus.cc:2463] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:25.303260 855 leader_election.cc:304] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
W20250112 02:52:25.303776 854 leader_election.cc:343] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.303905 854 leader_election.cc:304] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:25.304148 1190 raft_consensus.cc:2798] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:25.304239 1190 raft_consensus.cc:491] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:25.304329 1190 raft_consensus.cc:3054] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:25.304307 854 leader_election.cc:343] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.304646 1269 raft_consensus.cc:2743] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:25.304991 989 leader_election.cc:304] T 3f5b418c17084420b867498fda79fb0a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:25.305680 1302 raft_consensus.cc:2743] T 3f5b418c17084420b867498fda79fb0a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:25.307535 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 9163878cfeb64693a9c5493e812546af
I20250112 02:52:25.307437 1190 raft_consensus.cc:513] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.308233 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 9163878cfeb64693a9c5493e812546af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.308393 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_0188036e1bd0455ea82fd355934295fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:25.308703 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5601a9398c9a466582a98967c0e749e2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:25.308873 1190 leader_election.cc:290] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:25.308902 780 raft_consensus.cc:3054] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:25.308917 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0188036e1bd0455ea82fd355934295fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:25.309268 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 9163878cfeb64693a9c5493e812546af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9163878cfeb64693a9c5493e812546af already in progress: creating tablet
I20250112 02:52:25.310189 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5601a9398c9a466582a98967c0e749e2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:25.310294 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 9163878cfeb64693a9c5493e812546af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:25.310850 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 9163878cfeb64693a9c5493e812546af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9163878cfeb64693a9c5493e812546af already in progress: creating tablet
I20250112 02:52:25.312007 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 9163878cfeb64693a9c5493e812546af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:50844
W20250112 02:52:25.312559 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 9163878cfeb64693a9c5493e812546af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9163878cfeb64693a9c5493e812546af already in progress: creating tablet
I20250112 02:52:25.312914 780 raft_consensus.cc:2463] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
W20250112 02:52:25.313130 854 leader_election.cc:343] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.313335 855 leader_election.cc:304] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:25.313820 1190 raft_consensus.cc:2798] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:25.313957 1190 raft_consensus.cc:695] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:25.314234 1055 raft_consensus.cc:3054] T 3f5b418c17084420b867498fda79fb0a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:25.314245 1190 consensus_queue.cc:237] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.317520 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 9427951452104a7db765437493b21e90
I20250112 02:52:25.321841 526 master_service.cc:946] ReplaceTablet: received request to replace tablet aba1744138fe42ae9d097db01cac083d from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:25.322449 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 9427951452104a7db765437493b21e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.323962 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 9427951452104a7db765437493b21e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.324175 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 09886cd08c714f24a96e4df02bc0e6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:25.325480 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 9427951452104a7db765437493b21e90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9427951452104a7db765437493b21e90 already in progress: creating tablet
I20250112 02:52:25.327085 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 3567820f0d4349799b681fdfff0ebad8
I20250112 02:52:25.327523 521 catalog_manager.cc:6596] ReplaceTablet: tablet 3567820f0d4349799b681fdfff0ebad8 deleted and replaced by tablet 7a18132549374218893f35a4e4226b72
I20250112 02:52:25.328686 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 09886cd08c714f24a96e4df02bc0e6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.328884 759 tablet_service.cc:1514] Processing DeleteTablet for tablet 28abc02b1b21427eae06560b99bcc078 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.329025 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 235e9dd0964c4ea6af7d05ef58acae0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.329107 754 tablet_service.cc:1514] Processing DeleteTablet for tablet f05d3326545849b9af4519bc66c21069 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.329334 1171 tablet_replica.cc:331] T 09886cd08c714f24a96e4df02bc0e6c8 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:25.329581 1171 raft_consensus.cc:2238] T 09886cd08c714f24a96e4df02bc0e6c8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:25.329664 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 3567820f0d4349799b681fdfff0ebad8 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a18132549374218893f35a4e4226b72) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.329797 1171 raft_consensus.cc:2267] T 09886cd08c714f24a96e4df02bc0e6c8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:25.330292 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 9427951452104a7db765437493b21e90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9427951452104a7db765437493b21e90 already in progress: creating tablet
I20250112 02:52:25.330540 1171 ts_tablet_manager.cc:1905] T 09886cd08c714f24a96e4df02bc0e6c8 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:25.330543 511 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 09886cd08c714f24a96e4df02bc0e6c8 because the tablet was not found. No further retry: Not found: Tablet not found: 09886cd08c714f24a96e4df02bc0e6c8
I20250112 02:52:25.332724 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 3567820f0d4349799b681fdfff0ebad8 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a18132549374218893f35a4e4226b72) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:25.333187 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 3567820f0d4349799b681fdfff0ebad8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3567820f0d4349799b681fdfff0ebad8 already in progress: creating tablet
I20250112 02:52:25.334240 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 3567820f0d4349799b681fdfff0ebad8 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a18132549374218893f35a4e4226b72) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.334517 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 9427951452104a7db765437493b21e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:50844
W20250112 02:52:25.336215 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 3567820f0d4349799b681fdfff0ebad8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3567820f0d4349799b681fdfff0ebad8 already in progress: creating tablet
W20250112 02:52:25.336442 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 9427951452104a7db765437493b21e90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9427951452104a7db765437493b21e90 already in progress: creating tablet
I20250112 02:52:25.337224 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_679c07c6f25644c6a4136aa56dd5100a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.337705 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_679c07c6f25644c6a4136aa56dd5100a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.339253 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_91be8f5d6eb24e2ea86d927b7f062d1e" } new_table_name: "default.Table_43048dfe7bf24df5a7254bdb6854101e" modify_external_catalogs: true
I20250112 02:52:25.339740 1171 ts_tablet_manager.cc:1918] T 09886cd08c714f24a96e4df02bc0e6c8 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:25.339883 1171 log.cc:1198] T 09886cd08c714f24a96e4df02bc0e6c8 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/09886cd08c714f24a96e4df02bc0e6c8
I20250112 02:52:25.340209 1171 ts_tablet_manager.cc:1939] T 09886cd08c714f24a96e4df02bc0e6c8 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:25.341265 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_7ee8a748c60d439088423dcb284b17ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:25.341621 1171 tablet_replica.cc:331] T 28abc02b1b21427eae06560b99bcc078 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
W20250112 02:52:25.341773 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ee8a748c60d439088423dcb284b17ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.341856 1171 raft_consensus.cc:2238] T 28abc02b1b21427eae06560b99bcc078 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:25.342017 1171 raft_consensus.cc:2267] T 28abc02b1b21427eae06560b99bcc078 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:25.341353 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 09886cd08c714f24a96e4df02bc0e6c8 (table default.Table_f82a967adb9040f69df9732d0d228d8d [id=fce07a88937f4f5d8b14483e056bcb4d]) successfully deleted
I20250112 02:52:25.342715 1171 ts_tablet_manager.cc:1905] T 28abc02b1b21427eae06560b99bcc078 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:25.348946 1181 tablet_bootstrap.cc:492] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:25.349125 1181 ts_tablet_manager.cc:1397] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.056s user 0.001s sys 0.009s
I20250112 02:52:25.350057 1181 raft_consensus.cc:357] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.350340 1181 raft_consensus.cc:383] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.350430 1181 raft_consensus.cc:738] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.350754 1181 consensus_queue.cc:260] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.351382 1177 tablet_bootstrap.cc:492] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:25.351581 1177 ts_tablet_manager.cc:1397] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.051s user 0.014s sys 0.024s
I20250112 02:52:25.352922 1181 ts_tablet_manager.cc:1428] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250112 02:52:25.352346 1177 raft_consensus.cc:357] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.353395 1177 raft_consensus.cc:383] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.353497 1177 raft_consensus.cc:738] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.353730 1177 consensus_queue.cc:260] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.354936 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet cc62dcbf67e84e36ae1fbe127ccd4879
I20250112 02:52:25.355019 1181 tablet_bootstrap.cc:492] T 49650a419dfd49d5a1547770d3f2fe54 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:25.356503 1177 ts_tablet_manager.cc:1428] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 02:52:25.358083 1177 tablet_bootstrap.cc:492] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:25.353282 1309 tablet.cc:1721] T 3f5b418c17084420b867498fda79fb0a P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:25.360430 1177 tablet_bootstrap.cc:654] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.360400 754 tablet_service.cc:1514] Processing DeleteTablet for tablet cc62dcbf67e84e36ae1fbe127ccd4879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.360852 888 tablet_service.cc:1514] Processing DeleteTablet for tablet cc62dcbf67e84e36ae1fbe127ccd4879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:25.361583 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet cc62dcbf67e84e36ae1fbe127ccd4879 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc62dcbf67e84e36ae1fbe127ccd4879 already in progress: creating tablet
I20250112 02:52:25.362115 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet cc62dcbf67e84e36ae1fbe127ccd4879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.362645 1181 tablet_bootstrap.cc:654] T 49650a419dfd49d5a1547770d3f2fe54 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:25.362663 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet cc62dcbf67e84e36ae1fbe127ccd4879 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc62dcbf67e84e36ae1fbe127ccd4879 already in progress: creating tablet
W20250112 02:52:25.366168 1229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:52:25.366616 1229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:52:25.366797 1229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250112 02:52:25.368634 1181 tablet_bootstrap.cc:492] T 49650a419dfd49d5a1547770d3f2fe54 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:25.368767 1181 ts_tablet_manager.cc:1397] T 49650a419dfd49d5a1547770d3f2fe54 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.003s
I20250112 02:52:25.369896 1181 raft_consensus.cc:357] T 49650a419dfd49d5a1547770d3f2fe54 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.370110 1181 raft_consensus.cc:383] T 49650a419dfd49d5a1547770d3f2fe54 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.370199 1181 raft_consensus.cc:738] T 49650a419dfd49d5a1547770d3f2fe54 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.370447 1181 consensus_queue.cc:260] T 49650a419dfd49d5a1547770d3f2fe54 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.372637 1181 ts_tablet_manager.cc:1428] T 49650a419dfd49d5a1547770d3f2fe54 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250112 02:52:25.374122 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_f7982d4045f04157ae3aa0e8e8f6ffda" } new_table_name: "default.Table_2740664d25a8451b87667c0eaf8b4d1b" modify_external_catalogs: true
I20250112 02:52:25.374974 1181 tablet_bootstrap.cc:492] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:25.375061 1171 ts_tablet_manager.cc:1918] T 28abc02b1b21427eae06560b99bcc078 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:25.375174 1171 log.cc:1198] T 28abc02b1b21427eae06560b99bcc078 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/28abc02b1b21427eae06560b99bcc078
I20250112 02:52:25.376871 1181 tablet_bootstrap.cc:654] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.384061 1171 ts_tablet_manager.cc:1939] T 28abc02b1b21427eae06560b99bcc078 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:25.384877 1307 mvcc.cc:204] Tried to move back new op lower bound from 7113319813345308672 to 7113319811191287808. Current Snapshot: MvccSnapshot[applied={T|T < 7113319813345308672}]
I20250112 02:52:25.384864 1309 tablet.cc:1721] T 3f5b418c17084420b867498fda79fb0a P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:25.386220 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 28abc02b1b21427eae06560b99bcc078 (table default.Table_2bceb576abd94da4a7aca29c4b4fa89a [id=703dedbc15f549298f82cbbb62ee27d8]) successfully deleted
I20250112 02:52:25.386953 1181 tablet_bootstrap.cc:492] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:25.387137 1181 ts_tablet_manager.cc:1397] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20250112 02:52:25.387964 1181 raft_consensus.cc:357] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.388231 1181 raft_consensus.cc:383] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.388352 1181 raft_consensus.cc:738] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
W20250112 02:52:25.388679 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04f135b226774285a077e5806131d91b consensus state has no leader
I20250112 02:52:25.388664 1181 consensus_queue.cc:260] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.388870 1171 tablet_replica.cc:331] T 235e9dd0964c4ea6af7d05ef58acae0d P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:25.389055 1171 raft_consensus.cc:2238] T 235e9dd0964c4ea6af7d05ef58acae0d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:25.389216 1171 raft_consensus.cc:2267] T 235e9dd0964c4ea6af7d05ef58acae0d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:25.389113 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_8e00e020edc14465ad0e935be514d03d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.389596 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e00e020edc14465ad0e935be514d03d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.390079 1173 tablet_bootstrap.cc:492] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:25.390223 1173 ts_tablet_manager.cc:1397] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.173s user 0.019s sys 0.047s
I20250112 02:52:25.391023 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_66559fa571be4748b005f229c7089c09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:25.391243 1173 raft_consensus.cc:357] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.391454 1173 raft_consensus.cc:383] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 02:52:25.391492 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66559fa571be4748b005f229c7089c09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.391536 1173 raft_consensus.cc:738] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.391666 1181 ts_tablet_manager.cc:1428] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:25.391762 1173 consensus_queue.cc:260] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.392591 1173 ts_tablet_manager.cc:1428] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 02:52:25.392655 1181 tablet_bootstrap.cc:492] T 4bc067e895cb4c69af4dedc03533a276 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:25.393760 1173 tablet_bootstrap.cc:492] T 6834be02864543469056a928fafe28ca P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:25.395849 1173 tablet_bootstrap.cc:654] T 6834be02864543469056a928fafe28ca P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.396376 1171 ts_tablet_manager.cc:1905] T 235e9dd0964c4ea6af7d05ef58acae0d P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:25.396570 1177 tablet_bootstrap.cc:492] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:25.396719 1177 ts_tablet_manager.cc:1397] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20250112 02:52:25.397521 1177 raft_consensus.cc:357] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.397727 1177 raft_consensus.cc:383] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.397831 1177 raft_consensus.cc:738] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.398072 1177 consensus_queue.cc:260] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.398602 1177 ts_tablet_manager.cc:1428] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:25.398886 1177 tablet_bootstrap.cc:492] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:25.400525 1177 tablet_bootstrap.cc:654] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.402611 1181 tablet_bootstrap.cc:654] T 4bc067e895cb4c69af4dedc03533a276 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.402812 521 master_service.cc:946] ReplaceTablet: received request to replace tablet 1726d61cc36a4deb83c5d604e8daf4f0 from {username='slave'} at 127.0.0.1:40934
W20250112 02:52:25.403234 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb724feccc7f4fbc846fb6fa61be3385 consensus state has no leader
I20250112 02:52:25.405562 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_d3adef5e4ff541dabd0661d1c0997a71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.406225 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3adef5e4ff541dabd0661d1c0997a71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:25.407276 1229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20250112 02:52:25.412571 1173 tablet_bootstrap.cc:492] T 6834be02864543469056a928fafe28ca P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:25.412731 1173 ts_tablet_manager.cc:1397] T 6834be02864543469056a928fafe28ca P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.002s
W20250112 02:52:25.416620 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c639a2bcd4740e4be7b2b162004669a consensus state has no leader
I20250112 02:52:25.416947 526 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5601a9398c9a466582a98967c0e749e2 (table default.Table_3b1c87a5169e49b68fdf3d5c662a27bc [id=ed3672fa460a4aa2ae1d3b643ade43e6]). Expected version 2 got 0
I20250112 02:52:25.417018 522 master_service.cc:946] ReplaceTablet: received request to replace tablet b2d8ea2ef18a46508371b4789ab6f4fa from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:25.418743 518 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c2595782d66e4e4b890c94221b196daa (table default.Table_96a35d4e94284a49ba1a88d7d0acd3c1 [id=8559c454b17b4a00a60689a6a9b19f7d]). Expected version 1 got 0
I20250112 02:52:25.420542 888 tablet_service.cc:1467] Processing CreateTablet for tablet 1a8a178bafcb457f9b0e9587df560493 (DEFAULT_TABLE table=default.Table_3c1f0242d17443b5a758967cf82ca0ca [id=02303baf5298474488dbf02c831f3d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.421044 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a8a178bafcb457f9b0e9587df560493. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.421175 525 catalog_manager.cc:5526] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:25.421604 525 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5601a9398c9a466582a98967c0e749e2 (table default.Table_3b1c87a5169e49b68fdf3d5c662a27bc [id=ed3672fa460a4aa2ae1d3b643ade43e6]). Expected version 2 got 0
I20250112 02:52:25.421867 525 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet df6ccd04880442d0990c6cb0d27956bd (table default.Table_c667abf579554ea0af764de6515a18c6 [id=a8c0e9da5acc45c3ab7f67427f59e2b6]). Expected version 1 got 0
I20250112 02:52:25.424255 751 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0314aa03b1473ebbe05ce73adf7838 (DEFAULT_TABLE table=default.Table_9f27fd5786f84a8084cdf01f95f4519b [id=0e7297e807b545fa8482864ab59eeb41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.424692 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f0314aa03b1473ebbe05ce73adf7838. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.424968 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 49650a419dfd49d5a1547770d3f2fe54 with delete_type TABLET_DATA_DELETED (replaced by tablet d8eb0efc5d7f4c72bdfc7531a444d42a) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.425177 756 tablet_service.cc:1514] Processing DeleteTablet for tablet aa67bf25dc894d7ea96d247a22c14e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.425529 759 tablet_service.cc:1467] Processing CreateTablet for tablet 1a8a178bafcb457f9b0e9587df560493 (DEFAULT_TABLE table=default.Table_3c1f0242d17443b5a758967cf82ca0ca [id=02303baf5298474488dbf02c831f3d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.425904 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a8a178bafcb457f9b0e9587df560493. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.425832 1173 raft_consensus.cc:357] T 6834be02864543469056a928fafe28ca P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.426070 1173 raft_consensus.cc:383] T 6834be02864543469056a928fafe28ca P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.426148 1173 raft_consensus.cc:738] T 6834be02864543469056a928fafe28ca P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.426371 1173 consensus_queue.cc:260] T 6834be02864543469056a928fafe28ca P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.426913 1173 ts_tablet_manager.cc:1428] T 6834be02864543469056a928fafe28ca P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250112 02:52:25.427206 1173 tablet_bootstrap.cc:492] T e8220011788a480eb50ee4b670c45739 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:25.427747 750 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea075826b16475ca474e284fc6bc545 (DEFAULT_TABLE table=default.Table_e3301643d2294e8ea5f9c3c1bc366eb9 [id=c5005669bb5f4c029d673bd39cc1535d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.427906 757 tablet_service.cc:1467] Processing CreateTablet for tablet 49bb6d20cfa0453e90e048825114525e (DEFAULT_TABLE table=default.Table_f359341b73c14c60b859f06dc5f428a7 [id=bedc9786ee5b40a9bdcbd3cd0be38361]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.428059 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ea075826b16475ca474e284fc6bc545. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.428730 1173 tablet_bootstrap.cc:654] T e8220011788a480eb50ee4b670c45739 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.431121 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_952149cc55fa44fdbae173c5e83bc4f5" } new_table_name: "default.Table_2c6c931c403f4fb59c91e9ffabd5d3a6" modify_external_catalogs: true
I20250112 02:52:25.431449 753 tablet_service.cc:1467] Processing CreateTablet for tablet 8a12b225131b464494a9f02133d37aea (DEFAULT_TABLE table=default.Table_34c47be51dd0466ca34b2fdc2bba8919 [id=370e66eb490a48b5927d073f94481de0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.431820 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a12b225131b464494a9f02133d37aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.432287 898 tablet_service.cc:1467] Processing CreateTablet for tablet 02d0f67141dc46b6a87ca0ad080d5bfe (DEFAULT_TABLE table=default.Table_3a84890c771c4574a25d33543fe7217d [id=49a61d0f688b488da9a139f0a64d2834]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.432370 896 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0314aa03b1473ebbe05ce73adf7838 (DEFAULT_TABLE table=default.Table_9f27fd5786f84a8084cdf01f95f4519b [id=0e7297e807b545fa8482864ab59eeb41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.432709 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02d0f67141dc46b6a87ca0ad080d5bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.432912 899 tablet_service.cc:1467] Processing CreateTablet for tablet 8a12b225131b464494a9f02133d37aea (DEFAULT_TABLE table=default.Table_34c47be51dd0466ca34b2fdc2bba8919 [id=370e66eb490a48b5927d073f94481de0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.433239 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a12b225131b464494a9f02133d37aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.435695 760 tablet_service.cc:1467] Processing CreateTablet for tablet 2c03b5e612d64314b210c11686fdfa2a (DEFAULT_TABLE table=default.Table_f7f876c63e68490c805c0bfa3f0c25ea [id=5c4769bfcefd4fe0a0f3c2e315d50c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.435709 894 tablet_service.cc:1467] Processing CreateTablet for tablet 2c03b5e612d64314b210c11686fdfa2a (DEFAULT_TABLE table=default.Table_f7f876c63e68490c805c0bfa3f0c25ea [id=5c4769bfcefd4fe0a0f3c2e315d50c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.436049 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c03b5e612d64314b210c11686fdfa2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.436058 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c03b5e612d64314b210c11686fdfa2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.439314 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49bb6d20cfa0453e90e048825114525e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.440219 897 tablet_service.cc:1467] Processing CreateTablet for tablet ed7243c6329d4f11aa25eea2b7b67683 (DEFAULT_TABLE table=default.Table_f89a1b732e34437294f2a7cdc052b6ea [id=7d79cefc22aa44309a4fed4d8198c79f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.440548 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed7243c6329d4f11aa25eea2b7b67683. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.441339 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f0314aa03b1473ebbe05ce73adf7838. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:25.441538 1229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:52:25.441648 1229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 02:52:25.441731 1229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 02:52:25.443775 889 tablet_service.cc:1467] Processing CreateTablet for tablet 49bb6d20cfa0453e90e048825114525e (DEFAULT_TABLE table=default.Table_f359341b73c14c60b859f06dc5f428a7 [id=bedc9786ee5b40a9bdcbd3cd0be38361]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.444132 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49bb6d20cfa0453e90e048825114525e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.445914 1173 tablet_bootstrap.cc:492] T e8220011788a480eb50ee4b670c45739 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:25.446053 1173 ts_tablet_manager.cc:1397] T e8220011788a480eb50ee4b670c45739 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.007s
I20250112 02:52:25.446770 1173 raft_consensus.cc:357] T e8220011788a480eb50ee4b670c45739 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.446974 1173 raft_consensus.cc:383] T e8220011788a480eb50ee4b670c45739 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.447036 1173 raft_consensus.cc:738] T e8220011788a480eb50ee4b670c45739 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.447296 1173 consensus_queue.cc:260] T e8220011788a480eb50ee4b670c45739 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.449146 1173 ts_tablet_manager.cc:1428] T e8220011788a480eb50ee4b670c45739 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:25.449505 1173 tablet_bootstrap.cc:492] T aa67bf25dc894d7ea96d247a22c14e03 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:25.450102 890 tablet_service.cc:1467] Processing CreateTablet for tablet ea33e5c1a1fb46599c5836a36b130051 (DEFAULT_TABLE table=default.Table_7abcf14838c744109a08985ba6ce3e0d [id=5f25c42be8924b41be7f2f268d068e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.451778 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 1a8a178bafcb457f9b0e9587df560493 (DEFAULT_TABLE table=default.Table_3c1f0242d17443b5a758967cf82ca0ca [id=02303baf5298474488dbf02c831f3d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.452138 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a8a178bafcb457f9b0e9587df560493. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.453356 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_977c68d65cd443ef936f3dd77b445154"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.453841 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_977c68d65cd443ef936f3dd77b445154 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.454129 1173 tablet_bootstrap.cc:654] T aa67bf25dc894d7ea96d247a22c14e03 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.455081 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_05f78f72ee534992af566f4565d21eaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.455545 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05f78f72ee534992af566f4565d21eaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.455802 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0314aa03b1473ebbe05ce73adf7838 (DEFAULT_TABLE table=default.Table_9f27fd5786f84a8084cdf01f95f4519b [id=0e7297e807b545fa8482864ab59eeb41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.456141 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f0314aa03b1473ebbe05ce73adf7838. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.456789 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_218fd160176b4f55937d334142f2ecbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.457206 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_218fd160176b4f55937d334142f2ecbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.459385 1022 tablet_service.cc:1467] Processing CreateTablet for tablet ed7243c6329d4f11aa25eea2b7b67683 (DEFAULT_TABLE table=default.Table_f89a1b732e34437294f2a7cdc052b6ea [id=7d79cefc22aa44309a4fed4d8198c79f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.460270 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed7243c6329d4f11aa25eea2b7b67683. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.462651 522 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 5a8e1b96accd459eaed221ded4a453b6 (table default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94]). Expected version 2 got 0
I20250112 02:52:25.463150 522 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c2595782d66e4e4b890c94221b196daa (table default.Table_96a35d4e94284a49ba1a88d7d0acd3c1 [id=8559c454b17b4a00a60689a6a9b19f7d]). Expected version 1 got 0
I20250112 02:52:25.463656 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_83a9b08ba3c64e50bbb471dbe4a7ad70" } new_table_name: "default.Table_ab8c82be3f854aec990748d44c6b8a9d" modify_external_catalogs: true
I20250112 02:52:25.464825 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 49bb6d20cfa0453e90e048825114525e (DEFAULT_TABLE table=default.Table_f359341b73c14c60b859f06dc5f428a7 [id=bedc9786ee5b40a9bdcbd3cd0be38361]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.465152 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49bb6d20cfa0453e90e048825114525e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.465353 1024 tablet_service.cc:1467] Processing CreateTablet for tablet ea33e5c1a1fb46599c5836a36b130051 (DEFAULT_TABLE table=default.Table_7abcf14838c744109a08985ba6ce3e0d [id=5f25c42be8924b41be7f2f268d068e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.465598 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 2c03b5e612d64314b210c11686fdfa2a (DEFAULT_TABLE table=default.Table_f7f876c63e68490c805c0bfa3f0c25ea [id=5c4769bfcefd4fe0a0f3c2e315d50c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.465778 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea33e5c1a1fb46599c5836a36b130051. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.465371 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 02d0f67141dc46b6a87ca0ad080d5bfe (DEFAULT_TABLE table=default.Table_3a84890c771c4574a25d33543fe7217d [id=49a61d0f688b488da9a139f0a64d2834]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.467835 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea075826b16475ca474e284fc6bc545 (DEFAULT_TABLE table=default.Table_e3301643d2294e8ea5f9c3c1bc366eb9 [id=c5005669bb5f4c029d673bd39cc1535d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.466013 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c03b5e612d64314b210c11686fdfa2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.468289 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 8a12b225131b464494a9f02133d37aea (DEFAULT_TABLE table=default.Table_34c47be51dd0466ca34b2fdc2bba8919 [id=370e66eb490a48b5927d073f94481de0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.468590 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a12b225131b464494a9f02133d37aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.471283 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02d0f67141dc46b6a87ca0ad080d5bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.471968 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ea075826b16475ca474e284fc6bc545. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.475960 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_43048dfe7bf24df5a7254bdb6854101e" } modify_external_catalogs: true
I20250112 02:52:25.476145 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_43048dfe7bf24df5a7254bdb6854101e" } modify_external_catalogs: true
I20250112 02:52:25.478737 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 161fe26b9c0e4543b7aeb0021506d72a
I20250112 02:52:25.479249 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 1726d61cc36a4deb83c5d604e8daf4f0
I20250112 02:52:25.479203 758 tablet_service.cc:1467] Processing CreateTablet for tablet ed7243c6329d4f11aa25eea2b7b67683 (DEFAULT_TABLE table=default.Table_f89a1b732e34437294f2a7cdc052b6ea [id=7d79cefc22aa44309a4fed4d8198c79f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.479611 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed7243c6329d4f11aa25eea2b7b67683. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.479820 521 catalog_manager.cc:6596] ReplaceTablet: tablet 1726d61cc36a4deb83c5d604e8daf4f0 deleted and replaced by tablet b2ccb88108f24e36ba4b7ebf36398950
W20250112 02:52:25.481516 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 23e7a2e88c374e949ced92c557225cbf consensus state has no leader
I20250112 02:52:25.482004 892 tablet_service.cc:1514] Processing DeleteTablet for tablet f05d3326545849b9af4519bc66c21069 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.482367 892 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.482488 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 1726d61cc36a4deb83c5d604e8daf4f0 with delete_type TABLET_DATA_DELETED (replaced by tablet b2ccb88108f24e36ba4b7ebf36398950) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.482713 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 28abc02b1b21427eae06560b99bcc078 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.483000 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 71b5f80ed198491780cbe17e6cd3fa15 with delete_type TABLET_DATA_DELETED (replaced by tablet c7c530aa0aa64e47ae27d61f341d2bd9) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.483214 749 tablet_service.cc:1467] Processing CreateTablet for tablet 02d0f67141dc46b6a87ca0ad080d5bfe (DEFAULT_TABLE table=default.Table_3a84890c771c4574a25d33543fe7217d [id=49a61d0f688b488da9a139f0a64d2834]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.483336 886 tablet_service.cc:1514] Processing DeleteTablet for tablet d3e41e307df64109a41b0c1c44d3bda6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.483579 1175 tablet_replica.cc:331] T f05d3326545849b9af4519bc66c21069 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:25.483539 755 tablet_service.cc:1467] Processing CreateTablet for tablet ea33e5c1a1fb46599c5836a36b130051 (DEFAULT_TABLE table=default.Table_7abcf14838c744109a08985ba6ce3e0d [id=5f25c42be8924b41be7f2f268d068e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.483870 1175 raft_consensus.cc:2238] T f05d3326545849b9af4519bc66c21069 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:25.484097 1175 raft_consensus.cc:2267] T f05d3326545849b9af4519bc66c21069 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:25.484445 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 328e307a4b35445f92cbfab24f2ada48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.485258 1175 ts_tablet_manager.cc:1905] T f05d3326545849b9af4519bc66c21069 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:25.486158 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 161fe26b9c0e4543b7aeb0021506d72a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.487938 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02d0f67141dc46b6a87ca0ad080d5bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.490077 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 161fe26b9c0e4543b7aeb0021506d72a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.492156 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea33e5c1a1fb46599c5836a36b130051. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.493283 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 1726d61cc36a4deb83c5d604e8daf4f0 with delete_type TABLET_DATA_DELETED (replaced by tablet b2ccb88108f24e36ba4b7ebf36398950) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:25.498135 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 015412a5fa304453bd8d1be1c3cd7a46 consensus state has no leader
I20250112 02:52:25.504324 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet cb724feccc7f4fbc846fb6fa61be3385
I20250112 02:52:25.508569 898 tablet_service.cc:1514] Processing DeleteTablet for tablet cb724feccc7f4fbc846fb6fa61be3385 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.512143 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_f359341b73c14c60b859f06dc5f428a7" } modify_external_catalogs: true
I20250112 02:52:25.512365 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_f359341b73c14c60b859f06dc5f428a7" } modify_external_catalogs: true
I20250112 02:52:25.514811 526 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet df6ccd04880442d0990c6cb0d27956bd (table default.Table_c667abf579554ea0af764de6515a18c6 [id=a8c0e9da5acc45c3ab7f67427f59e2b6]). Expected version 1 got 0
I20250112 02:52:25.515856 526 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 96e201d798c94e448a78e1c8adf41724 (table default.Table_4c59f5d61aab4b6184f0d9a4e21f965f [id=95cdab3e555643fc990b77b2ee8f058a]). Expected version 1 got 0
I20250112 02:52:25.518805 756 tablet_service.cc:1467] Processing CreateTablet for tablet a85300a9aa1c4c50a3957a004bea7933 (DEFAULT_TABLE table=default.Table_679c07c6f25644c6a4136aa56dd5100a [id=3a4127ef6ed145d4bdbb562adadfbb5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.518966 760 tablet_service.cc:1514] Processing DeleteTablet for tablet cb724feccc7f4fbc846fb6fa61be3385 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.519371 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a85300a9aa1c4c50a3957a004bea7933. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.520031 520 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 96e201d798c94e448a78e1c8adf41724 (table default.Table_4c59f5d61aab4b6184f0d9a4e21f965f [id=95cdab3e555643fc990b77b2ee8f058a]). Expected version 1 got 0
I20250112 02:52:25.520915 886 tablet_service.cc:1467] Processing CreateTablet for tablet a85300a9aa1c4c50a3957a004bea7933 (DEFAULT_TABLE table=default.Table_679c07c6f25644c6a4136aa56dd5100a [id=3a4127ef6ed145d4bdbb562adadfbb5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.521261 898 tablet_service.cc:1467] Processing CreateTablet for tablet daef3a58605e40978f7389698b22265d (DEFAULT_TABLE table=default.Table_8e00e020edc14465ad0e935be514d03d [id=cf8528b7c33a47ce984da87382bef6a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.521953 885 tablet_service.cc:1467] Processing CreateTablet for tablet bb217afc08304c0ea35dee3120347ba1 (DEFAULT_TABLE table=default.Table_3888b008030c447d885c3961d470cee2 [id=da2e1bf15b21475085a3da6b16aecc4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.522892 887 tablet_service.cc:1467] Processing CreateTablet for tablet 7a18132549374218893f35a4e4226b72 (DEFAULT_TABLE table=default.Table_3fcca85a33054d7ab388f29bba85fe35 [id=18415100e94043e99f9884b853158ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.524088 891 tablet_service.cc:1467] Processing CreateTablet for tablet 79c6bdd89f5a44efb56d65ccb724308b (DEFAULT_TABLE table=default.Table_3bc5157a35854b91b0c087c3ea919965 [id=9e72fad141db43c09c7e4692769b6eba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.525254 754 tablet_service.cc:1467] Processing CreateTablet for tablet 4a57eb366f594f93a488c8250e195014 (DEFAULT_TABLE table=default.Table_7ee8a748c60d439088423dcb284b17ad [id=f836cdee435949d387336c9a52b6ed1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.527048 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_c667abf579554ea0af764de6515a18c6" } modify_external_catalogs: true
I20250112 02:52:25.527232 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_c667abf579554ea0af764de6515a18c6" } modify_external_catalogs: true
I20250112 02:52:25.527884 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 49650a419dfd49d5a1547770d3f2fe54 with delete_type TABLET_DATA_DELETED (replaced by tablet d8eb0efc5d7f4c72bdfc7531a444d42a) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.528483 895 tablet_service.cc:1467] Processing CreateTablet for tablet 4a57eb366f594f93a488c8250e195014 (DEFAULT_TABLE table=default.Table_7ee8a748c60d439088423dcb284b17ad [id=f836cdee435949d387336c9a52b6ed1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.529441 892 tablet_service.cc:1467] Processing CreateTablet for tablet 61593969d2234ce3a6089027b067d922 (DEFAULT_TABLE table=default.Table_0188036e1bd0455ea82fd355934295fc [id=b2f29883bc3844cf8b008345b849345c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.533003 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 2728a022ef0d4952966d397a32356503 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.534029 1172 tablet_replica.cc:331] T 2728a022ef0d4952966d397a32356503 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:25.534281 1172 raft_consensus.cc:2238] T 2728a022ef0d4952966d397a32356503 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:25.535557 1172 raft_consensus.cc:2267] T 2728a022ef0d4952966d397a32356503 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:25.536793 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 6834be02864543469056a928fafe28ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.538695 893 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea075826b16475ca474e284fc6bc545 (DEFAULT_TABLE table=default.Table_e3301643d2294e8ea5f9c3c1bc366eb9 [id=c5005669bb5f4c029d673bd39cc1535d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.539158 1172 ts_tablet_manager.cc:1905] T 2728a022ef0d4952966d397a32356503 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:25.539568 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet bdaec169cfb5471391fcd3050bb00431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.539707 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet e8220011788a480eb50ee4b670c45739 with delete_type TABLET_DATA_DELETED (replaced by tablet b885247d2cb449e0a29f6f5d81518589) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.539947 752 tablet_service.cc:1467] Processing CreateTablet for tablet 61593969d2234ce3a6089027b067d922 (DEFAULT_TABLE table=default.Table_0188036e1bd0455ea82fd355934295fc [id=b2f29883bc3844cf8b008345b849345c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.540266 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a57eb366f594f93a488c8250e195014. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.542037 750 tablet_service.cc:1467] Processing CreateTablet for tablet 7a18132549374218893f35a4e4226b72 (DEFAULT_TABLE table=default.Table_3fcca85a33054d7ab388f29bba85fe35 [id=18415100e94043e99f9884b853158ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.542594 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61593969d2234ce3a6089027b067d922. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.543547 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 1726d61cc36a4deb83c5d604e8daf4f0 with delete_type TABLET_DATA_DELETED (replaced by tablet b2ccb88108f24e36ba4b7ebf36398950) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.544108 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet cb724feccc7f4fbc846fb6fa61be3385 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.544292 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 161fe26b9c0e4543b7aeb0021506d72a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.544469 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c686fdc8896463c81c9f7c5f53a3da2) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.546694 1031 tablet_service.cc:1467] Processing CreateTablet for tablet daef3a58605e40978f7389698b22265d (DEFAULT_TABLE table=default.Table_8e00e020edc14465ad0e935be514d03d [id=cf8528b7c33a47ce984da87382bef6a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.548074 1022 tablet_service.cc:1467] Processing CreateTablet for tablet bb217afc08304c0ea35dee3120347ba1 (DEFAULT_TABLE table=default.Table_3888b008030c447d885c3961d470cee2 [id=da2e1bf15b21475085a3da6b16aecc4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.548540 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daef3a58605e40978f7389698b22265d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.547283 753 tablet_service.cc:1467] Processing CreateTablet for tablet daef3a58605e40978f7389698b22265d (DEFAULT_TABLE table=default.Table_8e00e020edc14465ad0e935be514d03d [id=cf8528b7c33a47ce984da87382bef6a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.550251 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb217afc08304c0ea35dee3120347ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.550261 760 tablet_service.cc:1467] Processing CreateTablet for tablet 79c6bdd89f5a44efb56d65ccb724308b (DEFAULT_TABLE table=default.Table_3bc5157a35854b91b0c087c3ea919965 [id=9e72fad141db43c09c7e4692769b6eba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.550022 751 tablet_service.cc:1467] Processing CreateTablet for tablet bb217afc08304c0ea35dee3120347ba1 (DEFAULT_TABLE table=default.Table_3888b008030c447d885c3961d470cee2 [id=da2e1bf15b21475085a3da6b16aecc4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.551219 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daef3a58605e40978f7389698b22265d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.552398 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a18132549374218893f35a4e4226b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.562716 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 4a57eb366f594f93a488c8250e195014 (DEFAULT_TABLE table=default.Table_7ee8a748c60d439088423dcb284b17ad [id=f836cdee435949d387336c9a52b6ed1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.563076 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a57eb366f594f93a488c8250e195014. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.564023 526 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_7abcf14838c744109a08985ba6ce3e0d" } modify_external_catalogs: true
I20250112 02:52:25.564424 526 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_7abcf14838c744109a08985ba6ce3e0d" } modify_external_catalogs: true
I20250112 02:52:25.567008 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79c6bdd89f5a44efb56d65ccb724308b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.567229 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb217afc08304c0ea35dee3120347ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.576196 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea33e5c1a1fb46599c5836a36b130051. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.577044 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 61593969d2234ce3a6089027b067d922 (DEFAULT_TABLE table=default.Table_0188036e1bd0455ea82fd355934295fc [id=b2f29883bc3844cf8b008345b849345c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.577430 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61593969d2234ce3a6089027b067d922. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.577715 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 79c6bdd89f5a44efb56d65ccb724308b (DEFAULT_TABLE table=default.Table_3bc5157a35854b91b0c087c3ea919965 [id=9e72fad141db43c09c7e4692769b6eba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.578125 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79c6bdd89f5a44efb56d65ccb724308b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.579159 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a18132549374218893f35a4e4226b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.580220 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_b3ed6c7a79ed43568b8743f8aea9f67e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.580757 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3ed6c7a79ed43568b8743f8aea9f67e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.582937 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 49bb6d20cfa0453e90e048825114525e
I20250112 02:52:25.585103 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet df6ccd04880442d0990c6cb0d27956bd
I20250112 02:52:25.586443 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ea075826b16475ca474e284fc6bc545. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.587034 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daef3a58605e40978f7389698b22265d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.587054 526 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet ea33e5c1a1fb46599c5836a36b130051
I20250112 02:52:25.587407 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79c6bdd89f5a44efb56d65ccb724308b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.588064 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a85300a9aa1c4c50a3957a004bea7933. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.588270 520 master_service.cc:946] ReplaceTablet: received request to replace tablet 09886cd08c714f24a96e4df02bc0e6c8 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:25.588366 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61593969d2234ce3a6089027b067d922. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.590381 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet df6ccd04880442d0990c6cb0d27956bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.590785 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 49bb6d20cfa0453e90e048825114525e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.593380 1190 raft_consensus.cc:491] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:25.593688 1302 raft_consensus.cc:491] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:25.593837 1302 raft_consensus.cc:513] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.594679 1302 leader_election.cc:290] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:25.595546 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb217afc08304c0ea35dee3120347ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.597689 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_b8256597ba354774ace25add22300525"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.598670 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8256597ba354774ace25add22300525 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.599951 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a57eb366f594f93a488c8250e195014. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.599891 1190 raft_consensus.cc:513] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.600950 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8e1b96accd459eaed221ded4a453b6" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:25.601038 1177 tablet_bootstrap.cc:492] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:25.601202 780 raft_consensus.cc:2388] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1e775296eb746db8b318f17d26820e6 in current term 1: Already voted for candidate fe36951a83fb4e57b26e187edbcbfe3d in this term.
I20250112 02:52:25.602429 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet ea33e5c1a1fb46599c5836a36b130051 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.603155 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_aa2df7ab46bb4ba5b5e774fe121c371b" } new_table_name: "default.Table_0bdc5f49d4ff4d6fb1290bc2dc0f70d9" modify_external_catalogs: true
I20250112 02:52:25.604825 1177 ts_tablet_manager.cc:1397] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.202s user 0.004s sys 0.012s
I20250112 02:52:25.604823 1229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42341,127.0.104.61:38771,127.0.104.60:39987
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:39987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=42831
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4135
ASAN enabled
I20250112 02:52:25.605726 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_3fec5f647ce141459d216dca71e5eb87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:25.605845 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8e1b96accd459eaed221ded4a453b6" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
W20250112 02:52:25.606204 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fec5f647ce141459d216dca71e5eb87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.606369 1190 leader_election.cc:290] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:25.606407 989 leader_election.cc:304] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:25.607005 1302 raft_consensus.cc:3054] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:25.607345 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_427e5903e2b04aeeab11c68a7e318d94" } new_table_name: "default.Table_c3651f53d9004655b02efca2428b3ead" modify_external_catalogs: true
I20250112 02:52:25.607635 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ee0e79dbb34b1ba9dcd39d36aa2ae7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:25.609380 854 leader_election.cc:343] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.611063 757 tablet_service.cc:1514] Processing DeleteTablet for tablet df6ccd04880442d0990c6cb0d27956bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.612047 899 tablet_service.cc:1514] Processing DeleteTablet for tablet ea33e5c1a1fb46599c5836a36b130051 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.612274 896 tablet_service.cc:1514] Processing DeleteTablet for tablet df6ccd04880442d0990c6cb0d27956bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.614331 1229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:52:25.615921 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 49bb6d20cfa0453e90e048825114525e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.619364 1229 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 02:52:25.620114 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ee0e79dbb34b1ba9dcd39d36aa2ae7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:25.620761 855 leader_election.cc:343] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.620983 855 leader_election.cc:304] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:25.621481 1190 raft_consensus.cc:2743] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:25.621888 758 tablet_service.cc:1514] Processing DeleteTablet for tablet ea33e5c1a1fb46599c5836a36b130051 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.623777 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_05f78f72ee534992af566f4565d21eaa" } new_table_name: "default.Table_feacb3254bbe4765a4e0a4908fbc694c" modify_external_catalogs: true
I20250112 02:52:25.625066 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_49b5979e0b174043b61f2f1d61a5b58c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.625543 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49b5979e0b174043b61f2f1d61a5b58c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.627954 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 49bb6d20cfa0453e90e048825114525e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.628751 1171 ts_tablet_manager.cc:1918] T 235e9dd0964c4ea6af7d05ef58acae0d P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:25.628949 1171 log.cc:1198] T 235e9dd0964c4ea6af7d05ef58acae0d P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/235e9dd0964c4ea6af7d05ef58acae0d
I20250112 02:52:25.630518 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_912cf75defb94f58b36cbfb445004436"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.631021 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_912cf75defb94f58b36cbfb445004436 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.631332 1171 ts_tablet_manager.cc:1939] T 235e9dd0964c4ea6af7d05ef58acae0d P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:25.632316 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 235e9dd0964c4ea6af7d05ef58acae0d (table default.Table_ae0485ac89ea47e7a8c19a07c5a63730 [id=e659e0cc87684ca79fc3b8241116742a]) successfully deleted
I20250112 02:52:25.628535 1177 raft_consensus.cc:357] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.636533 1177 raft_consensus.cc:383] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.636677 1177 raft_consensus.cc:738] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.637362 1177 consensus_queue.cc:260] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.640282 1181 tablet_bootstrap.cc:492] T 4bc067e895cb4c69af4dedc03533a276 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:25.640458 1181 ts_tablet_manager.cc:1397] T 4bc067e895cb4c69af4dedc03533a276 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.248s user 0.035s sys 0.065s
I20250112 02:52:25.641220 1181 raft_consensus.cc:357] T 4bc067e895cb4c69af4dedc03533a276 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.641403 1181 raft_consensus.cc:383] T 4bc067e895cb4c69af4dedc03533a276 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.641490 1181 raft_consensus.cc:738] T 4bc067e895cb4c69af4dedc03533a276 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.641719 1181 consensus_queue.cc:260] T 4bc067e895cb4c69af4dedc03533a276 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.642220 1181 ts_tablet_manager.cc:1428] T 4bc067e895cb4c69af4dedc03533a276 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 02:52:25.642509 1181 tablet_bootstrap.cc:492] T 3567820f0d4349799b681fdfff0ebad8 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:25.645531 1181 tablet_bootstrap.cc:654] T 3567820f0d4349799b681fdfff0ebad8 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.641311 1177 ts_tablet_manager.cc:1428] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.036s user 0.001s sys 0.001s
I20250112 02:52:25.644016 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bc067e895cb4c69af4dedc03533a276 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.647706 1171 tablet_replica.cc:331] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:25.648293 1171 raft_consensus.cc:2238] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:25.650238 1171 pending_rounds.cc:62] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:25.650558 1171 pending_rounds.cc:69] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319814105755648 op_type: NO_OP noop_request { }
I20250112 02:52:25.650758 1171 raft_consensus.cc:2883] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:25.650926 1171 raft_consensus.cc:2267] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:25.649061 1177 tablet_bootstrap.cc:492] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:25.651590 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bc067e895cb4c69af4dedc03533a276 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.653316 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_f9db9e236376446f8210cc4c9708ec6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.653851 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9db9e236376446f8210cc4c9708ec6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.658787 1171 ts_tablet_manager.cc:1905] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:25.661803 522 master_service.cc:946] ReplaceTablet: received request to replace tablet e9fc9a4f1e8944abb76dad6c2ad53d35 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:25.662168 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 7a18132549374218893f35a4e4226b72 (DEFAULT_TABLE table=default.Table_3fcca85a33054d7ab388f29bba85fe35 [id=18415100e94043e99f9884b853158ae1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:25.662269 1324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:52:25.665163 1322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:52:25.669245 753 tablet_service.cc:1467] Processing CreateTablet for tablet d659c5a4f7a446c6aaf2283f65b45106 (DEFAULT_TABLE table=default.Table_218fd160176b4f55937d334142f2ecbd [id=990e8109058d4dffbbd0ae23fc43783b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:25.670771 1330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:52:25.671070 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_8b4baeb21bec40af9997451f3eceddf1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.671638 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b4baeb21bec40af9997451f3eceddf1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.673527 1229 server_base.cc:1034] running on GCE node
I20250112 02:52:25.674578 1229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:52:25.677136 756 tablet_service.cc:1467] Processing CreateTablet for tablet 88d69632845547f588662dd56c8c194e (DEFAULT_TABLE table=default.Table_d3adef5e4ff541dabd0661d1c0997a71 [id=98b0d39d324c4b2b8e83dbb31776a5cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.678139 758 tablet_service.cc:1467] Processing CreateTablet for tablet 48634d6d78784fae98512b242e5476df (DEFAULT_TABLE table=default.Table_977c68d65cd443ef936f3dd77b445154 [id=fee2f9e65e314ce9a1fc1fb6dfcde317]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.679592 1023 tablet_service.cc:1467] Processing CreateTablet for tablet a85300a9aa1c4c50a3957a004bea7933 (DEFAULT_TABLE table=default.Table_679c07c6f25644c6a4136aa56dd5100a [id=3a4127ef6ed145d4bdbb562adadfbb5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.680079 1229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:52:25.680439 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a18132549374218893f35a4e4226b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.684852 1229 hybrid_clock.cc:648] HybridClock initialized: now 1736650345684727 us; error 37 us; skew 500 ppm
I20250112 02:52:25.685379 1229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:52:25.685516 1175 ts_tablet_manager.cc:1918] T f05d3326545849b9af4519bc66c21069 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:25.686867 1175 log.cc:1198] T f05d3326545849b9af4519bc66c21069 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/f05d3326545849b9af4519bc66c21069
I20250112 02:52:25.687310 1175 ts_tablet_manager.cc:1939] T f05d3326545849b9af4519bc66c21069 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
W20250112 02:52:25.690281 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09d06e2e8c13464db8cf801a292bdc99 consensus state has no leader
W20250112 02:52:25.691002 525 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c04b64bc2f8348a086ed3c4e59f09b16 consensus state has no leader
I20250112 02:52:25.691510 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_ab4d7c58924a4b8682b52637c23f125b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:25.691941 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a85300a9aa1c4c50a3957a004bea7933. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:25.692061 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab4d7c58924a4b8682b52637c23f125b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.692013 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_0c0859249ddd40ea8224fa9b119b3af3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.692519 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c0859249ddd40ea8224fa9b119b3af3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:25.693486 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b599e983ec04f53a440d1ab828d12ab consensus state has no leader
I20250112 02:52:25.694192 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_3bc5157a35854b91b0c087c3ea919965" } new_table_name: "default.Table_ad69cfacaf7b4050ab2d3d6ed237e7d0" modify_external_catalogs: true
I20250112 02:52:25.695493 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_96a35d4e94284a49ba1a88d7d0acd3c1" } modify_external_catalogs: true
I20250112 02:52:25.695683 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_96a35d4e94284a49ba1a88d7d0acd3c1" } modify_external_catalogs: true
I20250112 02:52:25.697072 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_45c4d5f155af43818f8fe32452dcf5d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.697554 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45c4d5f155af43818f8fe32452dcf5d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.699975 1302 raft_consensus.cc:2743] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:25.699920 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_2c6c931c403f4fb59c91e9ffabd5d3a6" } new_table_name: "default.Table_7a3faaceec4f4bd29b53455ec1228c80" modify_external_catalogs: true
I20250112 02:52:25.701161 1173 tablet_bootstrap.cc:492] T aa67bf25dc894d7ea96d247a22c14e03 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:25.701329 1173 ts_tablet_manager.cc:1397] T aa67bf25dc894d7ea96d247a22c14e03 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.252s user 0.033s sys 0.036s
I20250112 02:52:25.702836 1245 raft_consensus.cc:491] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:25.703029 1245 raft_consensus.cc:513] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.685688 1269 consensus_queue.cc:1035] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250112 02:52:25.705147 1181 tablet_bootstrap.cc:492] T 3567820f0d4349799b681fdfff0ebad8 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:25.706418 526 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_912cf75defb94f58b36cbfb445004436" } modify_external_catalogs: true
I20250112 02:52:25.706601 526 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_912cf75defb94f58b36cbfb445004436" } modify_external_catalogs: true
I20250112 02:52:25.705402 890 tablet_service.cc:1467] Processing CreateTablet for tablet 48634d6d78784fae98512b242e5476df (DEFAULT_TABLE table=default.Table_977c68d65cd443ef936f3dd77b445154 [id=fee2f9e65e314ce9a1fc1fb6dfcde317]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.708297 1181 ts_tablet_manager.cc:1397] T 3567820f0d4349799b681fdfff0ebad8 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.064s user 0.004s sys 0.009s
I20250112 02:52:25.709736 1181 raft_consensus.cc:357] T 3567820f0d4349799b681fdfff0ebad8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.710297 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48634d6d78784fae98512b242e5476df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.710891 1269 raft_consensus.cc:491] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:25.710955 887 tablet_service.cc:1467] Processing CreateTablet for tablet b2ccb88108f24e36ba4b7ebf36398950 (DEFAULT_TABLE table=default.Table_060115627d5f4bd786853027f080f260 [id=34b5449c89e248189460592f6eae98c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.712999 898 tablet_service.cc:1467] Processing CreateTablet for tablet 88d69632845547f588662dd56c8c194e (DEFAULT_TABLE table=default.Table_d3adef5e4ff541dabd0661d1c0997a71 [id=98b0d39d324c4b2b8e83dbb31776a5cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.713965 1173 raft_consensus.cc:357] T aa67bf25dc894d7ea96d247a22c14e03 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.713750 895 tablet_service.cc:1467] Processing CreateTablet for tablet 4b599e983ec04f53a440d1ab828d12ab (DEFAULT_TABLE table=default.Table_feacb3254bbe4765a4e0a4908fbc694c [id=f396d2dc8dbd47729f49f7511467d481]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.715107 893 tablet_service.cc:1467] Processing CreateTablet for tablet d659c5a4f7a446c6aaf2283f65b45106 (DEFAULT_TABLE table=default.Table_218fd160176b4f55937d334142f2ecbd [id=990e8109058d4dffbbd0ae23fc43783b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.716226 1181 raft_consensus.cc:383] T 3567820f0d4349799b681fdfff0ebad8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.716327 1181 raft_consensus.cc:738] T 3567820f0d4349799b681fdfff0ebad8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.717156 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet f05d3326545849b9af4519bc66c21069 (table default.Table_15a37175ae434dfc9638765263dfb60f [id=7acec972e5f54ed7b0ae6f3c8d317b68]) successfully deleted
I20250112 02:52:25.717417 892 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd48d0e68014e1aa0c44620527baec9 (DEFAULT_TABLE table=default.Table_66559fa571be4748b005f229c7089c09 [id=5431f5161ddf40d083b94b5fcb6b7561]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.718005 1181 consensus_queue.cc:260] T 3567820f0d4349799b681fdfff0ebad8 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.718556 1181 ts_tablet_manager.cc:1428] T 3567820f0d4349799b681fdfff0ebad8 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250112 02:52:25.720432 1269 raft_consensus.cc:513] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.720592 1181 tablet_bootstrap.cc:492] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:25.721297 1269 leader_election.cc:290] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:25.722143 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96e201d798c94e448a78e1c8adf41724" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:25.722373 779 raft_consensus.cc:2463] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:25.723713 855 leader_election.cc:304] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:25.725271 1173 raft_consensus.cc:383] T aa67bf25dc894d7ea96d247a22c14e03 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.726578 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2ccb88108f24e36ba4b7ebf36398950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.726616 1269 raft_consensus.cc:2798] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:25.726718 1269 raft_consensus.cc:491] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:25.726835 1269 raft_consensus.cc:3054] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:25.727646 1031 tablet_service.cc:1467] Processing CreateTablet for tablet d659c5a4f7a446c6aaf2283f65b45106 (DEFAULT_TABLE table=default.Table_218fd160176b4f55937d334142f2ecbd [id=990e8109058d4dffbbd0ae23fc43783b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.728488 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96e201d798c94e448a78e1c8adf41724" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:25.729013 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d659c5a4f7a446c6aaf2283f65b45106. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.729614 1269 raft_consensus.cc:513] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.730279 1175 tablet_replica.cc:331] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:25.731004 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5dd48d0e68014e1aa0c44620527baec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.732173 1181 tablet_bootstrap.cc:654] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.733361 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96e201d798c94e448a78e1c8adf41724" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:25.733530 779 raft_consensus.cc:3054] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:25.734382 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b599e983ec04f53a440d1ab828d12ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.734483 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 88d69632845547f588662dd56c8c194e (DEFAULT_TABLE table=default.Table_d3adef5e4ff541dabd0661d1c0997a71 [id=98b0d39d324c4b2b8e83dbb31776a5cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.734982 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88d69632845547f588662dd56c8c194e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.735442 1172 ts_tablet_manager.cc:1918] T 2728a022ef0d4952966d397a32356503 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:25.735584 1172 log.cc:1198] T 2728a022ef0d4952966d397a32356503 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/2728a022ef0d4952966d397a32356503
I20250112 02:52:25.735734 1269 leader_election.cc:290] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:25.735951 1175 raft_consensus.cc:2238] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:25.736142 1175 raft_consensus.cc:2267] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:25.736553 854 leader_election.cc:343] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.736617 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96e201d798c94e448a78e1c8adf41724" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:25.736814 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 48634d6d78784fae98512b242e5476df (DEFAULT_TABLE table=default.Table_977c68d65cd443ef936f3dd77b445154 [id=fee2f9e65e314ce9a1fc1fb6dfcde317]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.736945 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 3567820f0d4349799b681fdfff0ebad8 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a18132549374218893f35a4e4226b72) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:25.737180 854 leader_election.cc:343] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.737380 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d659c5a4f7a446c6aaf2283f65b45106. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:25.737710 512 consensus_peers.cc:487] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 -> Peer dd25f03f25df49178a5c53c636ecf304 (127.0.104.60:39987): Couldn't send request to peer dd25f03f25df49178a5c53c636ecf304. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39987: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250112 02:52:25.738333 1173 raft_consensus.cc:738] T aa67bf25dc894d7ea96d247a22c14e03 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.739001 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88d69632845547f588662dd56c8c194e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.743016 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet e9fc9a4f1e8944abb76dad6c2ad53d35
I20250112 02:52:25.736624 1175 ts_tablet_manager.cc:1905] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:25.743533 522 catalog_manager.cc:6596] ReplaceTablet: tablet e9fc9a4f1e8944abb76dad6c2ad53d35 deleted and replaced by tablet 5cc414d6420c41839621ab858067548b
I20250112 02:52:25.747193 1229 webserver.cc:458] Webserver started at http://127.0.104.60:42831/ using document root <none> and password file <none>
I20250112 02:52:25.750068 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d659c5a4f7a446c6aaf2283f65b45106. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.750576 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88d69632845547f588662dd56c8c194e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.750731 1229 fs_manager.cc:362] Metadata directory not provided
I20250112 02:52:25.751356 1229 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:52:25.752887 1245 leader_election.cc:290] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:25.753348 754 tablet_service.cc:1467] Processing CreateTablet for tablet b2ccb88108f24e36ba4b7ebf36398950 (DEFAULT_TABLE table=default.Table_060115627d5f4bd786853027f080f260 [id=34b5449c89e248189460592f6eae98c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.753727 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2ccb88108f24e36ba4b7ebf36398950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.755054 749 tablet_service.cc:1467] Processing CreateTablet for tablet 4b599e983ec04f53a440d1ab828d12ab (DEFAULT_TABLE table=default.Table_feacb3254bbe4765a4e0a4908fbc694c [id=f396d2dc8dbd47729f49f7511467d481]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.755455 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b599e983ec04f53a440d1ab828d12ab. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:25.758539 525 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 79c6bdd89f5a44efb56d65ccb724308b consensus state has no leader
I20250112 02:52:25.758901 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet c2595782d66e4e4b890c94221b196daa
I20250112 02:52:25.758888 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49650a419dfd49d5a1547770d3f2fe54" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
W20250112 02:52:25.760320 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 23e7a2e88c374e949ced92c557225cbf consensus state has no leader
I20250112 02:52:25.761061 526 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet b17b0d571f114647a89cdf34b7e3f244
I20250112 02:52:25.761582 919 raft_consensus.cc:2463] T 49650a419dfd49d5a1547770d3f2fe54 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:25.761909 885 tablet_service.cc:1514] Processing DeleteTablet for tablet c2595782d66e4e4b890c94221b196daa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.762266 713 leader_election.cc:304] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:25.762254 891 tablet_service.cc:1514] Processing DeleteTablet for tablet e9fc9a4f1e8944abb76dad6c2ad53d35 with delete_type TABLET_DATA_DELETED (replaced by tablet 5cc414d6420c41839621ab858067548b) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:25.762912 1245 raft_consensus.cc:2798] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:25.763028 1245 raft_consensus.cc:491] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:25.763139 1245 raft_consensus.cc:3054] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:25.765851 1245 raft_consensus.cc:513] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.767046 1245 leader_election.cc:290] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:25.767822 757 tablet_service.cc:1514] Processing DeleteTablet for tablet e9fc9a4f1e8944abb76dad6c2ad53d35 with delete_type TABLET_DATA_DELETED (replaced by tablet 5cc414d6420c41839621ab858067548b) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.768833 1229 fs_manager.cc:714] Time spent opening directory manager: real 0.014s user 0.004s sys 0.000s
I20250112 02:52:25.769538 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48634d6d78784fae98512b242e5476df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.772547 1181 tablet_bootstrap.cc:492] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:25.772707 1181 ts_tablet_manager.cc:1397] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.000s
I20250112 02:52:25.774632 1181 raft_consensus.cc:357] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.774861 1181 raft_consensus.cc:383] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.775276 1181 raft_consensus.cc:738] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.775882 1181 consensus_queue.cc:260] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.778574 760 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd48d0e68014e1aa0c44620527baec9 (DEFAULT_TABLE table=default.Table_66559fa571be4748b005f229c7089c09 [id=5431f5161ddf40d083b94b5fcb6b7561]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.779106 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5dd48d0e68014e1aa0c44620527baec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.779786 778 raft_consensus.cc:1270] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Refusing update from remote peer fe36951a83fb4e57b26e187edbcbfe3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 02:52:25.780370 1177 tablet_bootstrap.cc:654] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.782256 779 raft_consensus.cc:2463] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:25.783013 1190 consensus_queue.cc:1035] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:25.783912 1181 ts_tablet_manager.cc:1428] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250112 02:52:25.784026 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49650a419dfd49d5a1547770d3f2fe54" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:25.784370 855 leader_election.cc:304] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:25.786438 1181 tablet_bootstrap.cc:492] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:25.786849 526 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_66559fa571be4748b005f229c7089c09" } modify_external_catalogs: true
I20250112 02:52:25.787066 526 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_66559fa571be4748b005f229c7089c09" } modify_external_catalogs: true
I20250112 02:52:25.788026 521 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_cfef2f0eb8dd4927b6d0af0a8c3c739c" } modify_external_catalogs: true
I20250112 02:52:25.788251 521 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_cfef2f0eb8dd4927b6d0af0a8c3c739c" } modify_external_catalogs: true
I20250112 02:52:25.788918 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 4b599e983ec04f53a440d1ab828d12ab (DEFAULT_TABLE table=default.Table_feacb3254bbe4765a4e0a4908fbc694c [id=f396d2dc8dbd47729f49f7511467d481]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.790133 919 raft_consensus.cc:3054] T 49650a419dfd49d5a1547770d3f2fe54 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:25.791460 1269 raft_consensus.cc:2798] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:25.792443 1269 raft_consensus.cc:695] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:25.792722 1269 consensus_queue.cc:237] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.794338 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_b3ed6c7a79ed43568b8743f8aea9f67e" } modify_external_catalogs: true
I20250112 02:52:25.794520 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_b3ed6c7a79ed43568b8743f8aea9f67e" } modify_external_catalogs: true
I20250112 02:52:25.794543 1172 ts_tablet_manager.cc:1939] T 2728a022ef0d4952966d397a32356503 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:25.794890 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48634d6d78784fae98512b242e5476df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.795787 1055 raft_consensus.cc:1270] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Refusing update from remote peer fe36951a83fb4e57b26e187edbcbfe3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 02:52:25.796204 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b599e983ec04f53a440d1ab828d12ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.745959 1173 consensus_queue.cc:260] T aa67bf25dc894d7ea96d247a22c14e03 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.797869 1340 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:52:25.798192 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd48d0e68014e1aa0c44620527baec9 (DEFAULT_TABLE table=default.Table_66559fa571be4748b005f229c7089c09 [id=5431f5161ddf40d083b94b5fcb6b7561]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.799774 778 raft_consensus.cc:1270] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Refusing update from remote peer fe36951a83fb4e57b26e187edbcbfe3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 02:52:25.800304 1229 fs_manager.cc:730] Time spent opening block manager: real 0.027s user 0.004s sys 0.000s
I20250112 02:52:25.801668 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5dd48d0e68014e1aa0c44620527baec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.800475 1229 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/data,/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/wal
uuid: "dd25f03f25df49178a5c53c636ecf304"
format_stamp: "Formatted at 2025-01-12 02:52:20 on dist-test-slave-h743"
I20250112 02:52:25.801956 1229 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:52:25.804646 1022 tablet_service.cc:1467] Processing CreateTablet for tablet b2ccb88108f24e36ba4b7ebf36398950 (DEFAULT_TABLE table=default.Table_060115627d5f4bd786853027f080f260 [id=34b5449c89e248189460592f6eae98c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:25.808106 1190 consensus_queue.cc:1035] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20250112 02:52:25.808451 520 master_service.cc:946] ReplaceTablet: received request to replace tablet 1c639a2bcd4740e4be7b2b162004669a from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:25.809962 1054 raft_consensus.cc:1270] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Refusing update from remote peer fe36951a83fb4e57b26e187edbcbfe3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 02:52:25.813755 1245 raft_consensus.cc:491] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:25.814026 1245 raft_consensus.cc:513] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.814991 1245 leader_election.cc:290] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:25.815611 1173 ts_tablet_manager.cc:1428] T aa67bf25dc894d7ea96d247a22c14e03 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.098s user 0.001s sys 0.002s
I20250112 02:52:25.815605 887 tablet_service.cc:1514] Processing DeleteTablet for tablet fb4c004397ee480d897e8b125d248829 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:25.817848 854 consensus_peers.cc:487] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:25.817180 1190 consensus_queue.cc:1035] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250112 02:52:25.818393 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc067e895cb4c69af4dedc03533a276" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:25.819304 918 raft_consensus.cc:2463] T 4bc067e895cb4c69af4dedc03533a276 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:25.819823 713 leader_election.cc:304] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:25.820215 1190 raft_consensus.cc:491] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:25.820358 1190 raft_consensus.cc:513] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.822613 1245 raft_consensus.cc:2798] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:25.822710 1245 raft_consensus.cc:491] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:25.822798 1245 raft_consensus.cc:3054] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:25.825872 1245 raft_consensus.cc:513] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.825969 525 master_service.cc:946] ReplaceTablet: received request to replace tablet 1726d61cc36a4deb83c5d604e8daf4f0 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:25.826460 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b5f80ed198491780cbe17e6cd3fa15" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:25.826767 1245 leader_election.cc:290] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:25.827615 1190 leader_election.cc:290] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:25.827734 855 leader_election.cc:343] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.827046 1172 tablet_replica.cc:331] T 6834be02864543469056a928fafe28ca P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:25.827644 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b5f80ed198491780cbe17e6cd3fa15" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:25.828363 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc067e895cb4c69af4dedc03533a276" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:25.828547 918 raft_consensus.cc:3054] T 4bc067e895cb4c69af4dedc03533a276 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:25.829584 754 tablet_service.cc:1514] Processing DeleteTablet for tablet c2595782d66e4e4b890c94221b196daa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:25.830065 1190 consensus_queue.cc:1035] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250112 02:52:25.831449 1181 tablet_bootstrap.cc:654] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:25.842432 854 leader_election.cc:343] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.842492 919 raft_consensus.cc:2463] T 49650a419dfd49d5a1547770d3f2fe54 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:25.842962 713 leader_election.cc:304] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:25.843268 1294 raft_consensus.cc:491] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:25.843413 1294 raft_consensus.cc:513] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.843887 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2ccb88108f24e36ba4b7ebf36398950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:25.844246 1294 leader_election.cc:290] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:25.844406 1294 raft_consensus.cc:2798] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:25.844512 1294 raft_consensus.cc:695] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:25.844727 1294 consensus_queue.cc:237] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.850473 1302 raft_consensus.cc:491] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:25.850628 1302 raft_consensus.cc:513] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.851248 1171 ts_tablet_manager.cc:1918] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:25.851365 1171 log.cc:1198] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/f05d3326545849b9af4519bc66c21069
I20250112 02:52:25.851873 1171 ts_tablet_manager.cc:1939] T f05d3326545849b9af4519bc66c21069 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:25.852663 1302 leader_election.cc:290] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:25.853056 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet f05d3326545849b9af4519bc66c21069 (table default.Table_15a37175ae434dfc9638765263dfb60f [id=7acec972e5f54ed7b0ae6f3c8d317b68]) successfully deleted
W20250112 02:52:25.853173 991 leader_election.cc:343] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: bdaec169cfb5471391fcd3050bb00431
I20250112 02:52:25.853801 918 raft_consensus.cc:2463] T 4bc067e895cb4c69af4dedc03533a276 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:25.854378 713 leader_election.cc:304] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:25.854766 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6ccd04880442d0990c6cb0d27956bd" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:25.855048 919 raft_consensus.cc:2463] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:25.857498 1175 ts_tablet_manager.cc:1918] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:25.857865 1294 raft_consensus.cc:2798] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:25.858062 1294 raft_consensus.cc:695] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:25.858363 1294 consensus_queue.cc:237] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.859977 1175 log.cc:1198] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/d6ee0e79dbb34b1ba9dcd39d36aa2ae7
I20250112 02:52:25.860459 1175 ts_tablet_manager.cc:1939] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:25.850728 854 leader_election.cc:304] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:25.861084 1190 raft_consensus.cc:2743] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:25.862725 989 leader_election.cc:343] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: bdaec169cfb5471391fcd3050bb00431
I20250112 02:52:25.862864 989 leader_election.cc:304] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:25.863368 1302 raft_consensus.cc:2743] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:25.864015 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 (table default.Table_c360a95859e5448f9af5f90629a898e4 [id=08b0c7e4e0c54ec4a215a390e83a1cf6]) successfully deleted
I20250112 02:52:25.864099 1175 tablet_replica.cc:331] T 28abc02b1b21427eae06560b99bcc078 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:25.864310 1175 raft_consensus.cc:2238] T 28abc02b1b21427eae06560b99bcc078 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:25.864460 1175 raft_consensus.cc:2267] T 28abc02b1b21427eae06560b99bcc078 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:25.864503 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 1726d61cc36a4deb83c5d604e8daf4f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1726d61cc36a4deb83c5d604e8daf4f0 already in progress: creating tablet
I20250112 02:52:25.865021 1175 ts_tablet_manager.cc:1905] T 28abc02b1b21427eae06560b99bcc078 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:25.866096 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_548027a9e05c4376b024df028f14e86d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.866683 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_548027a9e05c4376b024df028f14e86d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.869021 517 catalog_manager.cc:5526] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:25.869458 517 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 96e201d798c94e448a78e1c8adf41724 (table default.Table_4c59f5d61aab4b6184f0d9a4e21f965f [id=95cdab3e555643fc990b77b2ee8f058a]). Expected version 1 got 0
I20250112 02:52:25.869585 1173 tablet_bootstrap.cc:492] T 4bc067e895cb4c69af4dedc03533a276 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:25.874260 1172 raft_consensus.cc:2238] T 6834be02864543469056a928fafe28ca P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:25.874425 1172 raft_consensus.cc:2267] T 6834be02864543469056a928fafe28ca P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:25.875505 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49650a419dfd49d5a1547770d3f2fe54" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:25.877972 1181 tablet_bootstrap.cc:492] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:25.878253 1172 ts_tablet_manager.cc:1905] T 6834be02864543469056a928fafe28ca P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:25.879104 1181 ts_tablet_manager.cc:1397] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.093s user 0.002s sys 0.009s
I20250112 02:52:25.879958 1181 raft_consensus.cc:357] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.880205 1181 raft_consensus.cc:383] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.880275 1181 raft_consensus.cc:738] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.880482 1181 consensus_queue.cc:260] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.880929 1181 ts_tablet_manager.cc:1428] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:25.881214 1181 tablet_bootstrap.cc:492] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:25.882817 1181 tablet_bootstrap.cc:654] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.886895 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 2728a022ef0d4952966d397a32356503 (table default.Table_6d358fc697c74529822f0b2b70321a02 [id=9b36703d61a046799cb709c7618b1fbd]) successfully deleted
I20250112 02:52:25.885339 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49650a419dfd49d5a1547770d3f2fe54" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:25.892752 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_d327837707034a8297f472fd6a4f7af2" } new_table_name: "default.Table_63649e9607394f4bb72a3dd6e6d0dd69" modify_external_catalogs: true
I20250112 02:52:25.892864 1352 tablet.cc:1721] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:25.891543 713 leader_election.cc:304] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:25.895020 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_3e58cdf6ca254711b8071a2899d2402f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.895449 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e58cdf6ca254711b8071a2899d2402f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.895907 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_7d54fa05bdbf4ff3b5cce201222e62de" } new_table_name: "default.Table_045e5b104dc64217aa39c22534007cab" modify_external_catalogs: true
I20250112 02:52:25.897939 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_bac47d0dd2ab473babf687f5df1691c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:25.898361 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bac47d0dd2ab473babf687f5df1691c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:25.898617 1294 raft_consensus.cc:2798] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:25.899294 1294 raft_consensus.cc:491] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:25.899421 1294 raft_consensus.cc:3054] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:25.903424 1177 tablet_bootstrap.cc:492] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:25.907975 1352 tablet.cc:1721] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 02:52:25.908741 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 3567820f0d4349799b681fdfff0ebad8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3567820f0d4349799b681fdfff0ebad8 already in progress: creating tablet
I20250112 02:52:25.917690 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet e9fc9a4f1e8944abb76dad6c2ad53d35 with delete_type TABLET_DATA_DELETED (replaced by tablet 5cc414d6420c41839621ab858067548b) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.891995 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc067e895cb4c69af4dedc03533a276" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:25.921795 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet c2595782d66e4e4b890c94221b196daa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.922335 1342 raft_consensus.cc:491] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:25.922468 1342 raft_consensus.cc:513] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.923101 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet aa67bf25dc894d7ea96d247a22c14e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:25.923359 1342 leader_election.cc:290] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:25.924650 1229 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:52:25.925724 1229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:52:25.925807 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e41e307df64109a41b0c1c44d3bda6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:25.926014 1229 kserver.cc:163] Server-wide thread pool size limit: 3276
W20250112 02:52:25.926266 855 leader_election.cc:343] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.890149 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc067e895cb4c69af4dedc03533a276" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:25.930253 714 leader_election.cc:343] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:25.930658 714 leader_election.cc:343] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.930756 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e41e307df64109a41b0c1c44d3bda6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:25.931025 714 leader_election.cc:343] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:25.931347 854 leader_election.cc:343] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.931458 854 leader_election.cc:304] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:25.919512 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6ccd04880442d0990c6cb0d27956bd" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:25.931932 1173 tablet_bootstrap.cc:654] T 4bc067e895cb4c69af4dedc03533a276 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.932050 1352 tablet.cc:1721] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 02:52:25.932677 714 leader_election.cc:343] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:25.933323 714 leader_election.cc:343] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.933775 1342 raft_consensus.cc:2743] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:25.933940 1353 tablet.cc:1721] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:25.937000 1172 ts_tablet_manager.cc:1918] T 6834be02864543469056a928fafe28ca P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:25.937134 1172 log.cc:1198] T 6834be02864543469056a928fafe28ca P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/6834be02864543469056a928fafe28ca
I20250112 02:52:25.937481 1172 ts_tablet_manager.cc:1939] T 6834be02864543469056a928fafe28ca P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:25.937909 1350 raft_consensus.cc:491] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:25.938088 1350 raft_consensus.cc:513] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:25.939280 1350 leader_election.cc:290] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:25.940001 713 leader_election.cc:343] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: aa67bf25dc894d7ea96d247a22c14e03
I20250112 02:52:25.940220 1294 raft_consensus.cc:513] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.940490 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa67bf25dc894d7ea96d247a22c14e03" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:25.940742 1049 raft_consensus.cc:2463] T aa67bf25dc894d7ea96d247a22c14e03 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:25.940896 1294 leader_election.cc:290] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:25.941381 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6ccd04880442d0990c6cb0d27956bd" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:25.941576 919 raft_consensus.cc:3054] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:25.942062 1358 tablet.cc:1721] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:25.942468 1177 ts_tablet_manager.cc:1397] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.255s user 0.004s sys 0.000s
I20250112 02:52:25.943186 1177 raft_consensus.cc:357] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.943333 1177 raft_consensus.cc:383] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.943394 1177 raft_consensus.cc:738] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.943632 1177 consensus_queue.cc:260] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.944142 1177 ts_tablet_manager.cc:1428] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:25.944766 919 raft_consensus.cc:2463] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:25.945075 1177 tablet_bootstrap.cc:492] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:25.945381 713 leader_election.cc:304] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:25.947005 714 leader_election.cc:304] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:25.947305 1245 raft_consensus.cc:2798] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:25.947432 1245 raft_consensus.cc:695] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:25.947667 1245 consensus_queue.cc:237] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.950217 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6ccd04880442d0990c6cb0d27956bd" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:25.950762 1350 raft_consensus.cc:2798] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:25.950932 1172 tablet_replica.cc:331] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:25.960136 1342 raft_consensus.cc:491] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:25.960309 1342 raft_consensus.cc:513] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.961548 1342 leader_election.cc:290] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:25.965454 1172 raft_consensus.cc:2238] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:25.965745 1172 raft_consensus.cc:2267] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:25.966470 1172 ts_tablet_manager.cc:1905] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:25.951002 714 leader_election.cc:343] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.957151 1362 tablet.cc:1721] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:25.964682 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9533ea72f6420bb55a53cdfc9125f7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:25.972386 854 leader_election.cc:343] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.961839 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9533ea72f6420bb55a53cdfc9125f7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:25.965986 1360 tablet.cc:1721] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 02:52:25.973117 855 leader_election.cc:343] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:25.973292 855 leader_election.cc:304] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:25.973567 1342 raft_consensus.cc:2743] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:25.975643 1173 tablet_bootstrap.cc:492] T 4bc067e895cb4c69af4dedc03533a276 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:25.975801 1173 ts_tablet_manager.cc:1397] T 4bc067e895cb4c69af4dedc03533a276 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.135s user 0.004s sys 0.007s
I20250112 02:52:25.976320 1350 raft_consensus.cc:491] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:25.976629 1350 raft_consensus.cc:3054] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:25.976902 1173 raft_consensus.cc:357] T 4bc067e895cb4c69af4dedc03533a276 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.977111 1173 raft_consensus.cc:383] T 4bc067e895cb4c69af4dedc03533a276 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:25.977198 1173 raft_consensus.cc:738] T 4bc067e895cb4c69af4dedc03533a276 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:25.977464 1173 consensus_queue.cc:260] T 4bc067e895cb4c69af4dedc03533a276 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:25.978282 1173 ts_tablet_manager.cc:1428] T 4bc067e895cb4c69af4dedc03533a276 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:25.978824 1173 tablet_bootstrap.cc:492] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:25.979040 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet cc62dcbf67e84e36ae1fbe127ccd4879 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc62dcbf67e84e36ae1fbe127ccd4879 already in progress: creating tablet
I20250112 02:52:25.979513 1171 tablet_replica.cc:331] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:25.979768 1171 raft_consensus.cc:2238] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:25.980119 1171 pending_rounds.cc:62] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:25.980263 1171 pending_rounds.cc:69] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319816581910528 op_type: NO_OP noop_request { }
I20250112 02:52:25.980590 1171 raft_consensus.cc:2883] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:25.980782 1171 raft_consensus.cc:2267] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:25.981609 1171 ts_tablet_manager.cc:1905] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:25.982409 1173 tablet_bootstrap.cc:654] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:25.998143 1175 ts_tablet_manager.cc:1918] T 28abc02b1b21427eae06560b99bcc078 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:25.998378 1175 log.cc:1198] T 28abc02b1b21427eae06560b99bcc078 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/28abc02b1b21427eae06560b99bcc078
I20250112 02:52:25.998797 1175 ts_tablet_manager.cc:1939] T 28abc02b1b21427eae06560b99bcc078 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:26.000144 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 28abc02b1b21427eae06560b99bcc078 (table default.Table_2bceb576abd94da4a7aca29c4b4fa89a [id=703dedbc15f549298f82cbbb62ee27d8]) successfully deleted
I20250112 02:52:26.000582 1175 tablet_replica.cc:331] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:26.001854 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 6834be02864543469056a928fafe28ca (table default.Table_5fe703bbcb87402b87a82e1441569d53 [id=a2449caa011d43bfb54aea196629c9f1]) successfully deleted
I20250112 02:52:26.003993 1177 tablet_bootstrap.cc:654] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.005064 525 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5a8e1b96accd459eaed221ded4a453b6 (table default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94]). Expected version 2 got 0
I20250112 02:52:26.007242 525 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 5a8e1b96accd459eaed221ded4a453b6 (table default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94]). Expected version 2 got 0
I20250112 02:52:26.008594 1350 raft_consensus.cc:513] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.009433 1350 leader_election.cc:290] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:26.017592 713 leader_election.cc:343] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: aa67bf25dc894d7ea96d247a22c14e03
I20250112 02:52:26.020435 1177 tablet_bootstrap.cc:492] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:26.020588 1177 ts_tablet_manager.cc:1397] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.000s
I20250112 02:52:26.021093 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa67bf25dc894d7ea96d247a22c14e03" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:26.021312 1049 raft_consensus.cc:3054] T aa67bf25dc894d7ea96d247a22c14e03 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:26.023653 1177 raft_consensus.cc:357] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.023890 1177 raft_consensus.cc:383] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.023957 1177 raft_consensus.cc:738] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.023919 1049 raft_consensus.cc:2463] T aa67bf25dc894d7ea96d247a22c14e03 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:26.024183 1177 consensus_queue.cc:260] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.024639 1177 ts_tablet_manager.cc:1428] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:26.024909 1177 tablet_bootstrap.cc:492] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:26.025333 1362 tablet.cc:1721] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:26.025674 1172 ts_tablet_manager.cc:1918] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:26.025830 1172 log.cc:1198] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/bdaec169cfb5471391fcd3050bb00431
I20250112 02:52:26.026175 1172 ts_tablet_manager.cc:1939] T bdaec169cfb5471391fcd3050bb00431 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:26.027079 1172 tablet_replica.cc:331] T e8220011788a480eb50ee4b670c45739 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:26.027139 1177 tablet_bootstrap.cc:654] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.027272 1172 raft_consensus.cc:2238] T e8220011788a480eb50ee4b670c45739 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:26.028476 714 leader_election.cc:304] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:26.030622 1171 ts_tablet_manager.cc:1918] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:26.030783 1171 log.cc:1198] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/49650a419dfd49d5a1547770d3f2fe54
I20250112 02:52:26.031119 1171 ts_tablet_manager.cc:1939] T 49650a419dfd49d5a1547770d3f2fe54 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:26.032078 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 49650a419dfd49d5a1547770d3f2fe54 (table default.Table_d561e1740cc545dc8aa4babb76b45c37 [id=de27aeff6b3a49b3956498ea5015ff33]) successfully deleted
I20250112 02:52:26.027431 1172 raft_consensus.cc:2267] T e8220011788a480eb50ee4b670c45739 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.033957 1172 ts_tablet_manager.cc:1905] T e8220011788a480eb50ee4b670c45739 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.034838 1350 raft_consensus.cc:2798] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:26.034984 1350 raft_consensus.cc:695] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:26.035254 1350 consensus_queue.cc:237] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.043903 1171 tablet_replica.cc:331] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:26.046077 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet bdaec169cfb5471391fcd3050bb00431 (table default.Table_7d31752a7e7246ad9806310039f73c14 [id=62176e03ce0b4ced96e5083fa32b1826]) successfully deleted
W20250112 02:52:26.046317 512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39987: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250112 02:52:26.049773 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f8c01b80fc8242d98fdfbf767212144d consensus state has no leader
W20250112 02:52:26.053283 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet abfaad67d5e34c0a8338dccc0ef05466 consensus state has no leader
I20250112 02:52:26.053711 523 master_service.cc:946] ReplaceTablet: received request to replace tablet 6913b0ac9ceb4dbc9cdf591f159134f7 from {username='slave'} at 127.0.0.1:40934
W20250112 02:52:26.054255 513 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 04f135b226774285a077e5806131d91b (table default.Table_d1e5cef5fa3e4d90bada176f80bd4180 [id=f93b5bb22ec1424ea047383ed9713618], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 04f135b226774285a077e5806131d91b consensus state has no leader
I20250112 02:52:26.054677 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_2e35ffa5571d416d9b68443869009ab4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:26.054821 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_bee5a719ea4b44b78254bae566a0ba92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:26.054971 525 catalog_manager.cc:6071] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94] alter complete (version 2)
W20250112 02:52:26.055195 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e35ffa5571d416d9b68443869009ab4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:26.055197 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bee5a719ea4b44b78254bae566a0ba92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.055790 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_c734b496a0634288b6a93d8007262510"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.056385 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c734b496a0634288b6a93d8007262510 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.056471 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_10e019e7a9244dbeac8bd2a55f618830" } new_table_name: "default.Table_83177e7dc14645dbb0e5ef573e80e72f" modify_external_catalogs: true
I20250112 02:52:26.066426 1302 raft_consensus.cc:491] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:26.066601 1302 raft_consensus.cc:513] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.067354 1302 leader_election.cc:290] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:26.069008 991 leader_election.cc:343] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: 325e43fdf2fc4eacb48a5227e2ff3bf7
W20250112 02:52:26.069794 989 leader_election.cc:343] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 325e43fdf2fc4eacb48a5227e2ff3bf7
I20250112 02:52:26.069938 989 leader_election.cc:304] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:26.070374 1302 raft_consensus.cc:2743] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:26.072136 1172 ts_tablet_manager.cc:1918] T e8220011788a480eb50ee4b670c45739 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:26.074983 1172 log.cc:1198] T e8220011788a480eb50ee4b670c45739 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/e8220011788a480eb50ee4b670c45739
I20250112 02:52:26.076092 1245 raft_consensus.cc:491] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:26.076269 1245 raft_consensus.cc:513] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.076668 1172 ts_tablet_manager.cc:1939] T e8220011788a480eb50ee4b670c45739 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:26.077004 1245 leader_election.cc:290] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:26.080746 1172 tablet_replica.cc:331] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:26.080973 1172 raft_consensus.cc:2238] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:26.081341 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3567820f0d4349799b681fdfff0ebad8" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:26.082232 714 leader_election.cc:343] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.078088 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3567820f0d4349799b681fdfff0ebad8" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:26.082702 919 raft_consensus.cc:2463] T 3567820f0d4349799b681fdfff0ebad8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:26.083149 713 leader_election.cc:304] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:26.083768 1245 raft_consensus.cc:2798] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:26.081166 1172 raft_consensus.cc:2267] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.083885 1245 raft_consensus.cc:491] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:26.083972 1245 raft_consensus.cc:3054] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:26.084934 1172 ts_tablet_manager.cc:1905] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.089586 1245 raft_consensus.cc:513] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.090759 1269 raft_consensus.cc:491] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:26.090909 1269 raft_consensus.cc:513] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.091524 1269 leader_election.cc:290] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:26.091452 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3567820f0d4349799b681fdfff0ebad8" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:26.091696 919 raft_consensus.cc:3054] T 3567820f0d4349799b681fdfff0ebad8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:26.092463 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9427951452104a7db765437493b21e90" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:26.092465 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9427951452104a7db765437493b21e90" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:26.092927 855 leader_election.cc:343] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:26.093066 854 leader_election.cc:343] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.093236 854 leader_election.cc:304] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:26.093638 1269 raft_consensus.cc:2743] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:26.094902 919 raft_consensus.cc:2463] T 3567820f0d4349799b681fdfff0ebad8 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:26.095350 713 leader_election.cc:304] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:26.095655 1294 raft_consensus.cc:2798] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:26.096432 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3567820f0d4349799b681fdfff0ebad8" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:26.097218 714 leader_election.cc:343] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.098032 1294 raft_consensus.cc:695] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:26.098327 1294 consensus_queue.cc:237] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.100600 1245 leader_election.cc:290] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:26.102425 1172 ts_tablet_manager.cc:1918] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:26.102584 1172 log.cc:1198] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/325e43fdf2fc4eacb48a5227e2ff3bf7
I20250112 02:52:26.104523 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet e8220011788a480eb50ee4b670c45739 (table default.Table_edaa8b50dfa8463e99e05ab8d1d89bc3 [id=a6a18be9a78e4959b7d272c447f6fc25]) successfully deleted
W20250112 02:52:26.104794 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 1726d61cc36a4deb83c5d604e8daf4f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1726d61cc36a4deb83c5d604e8daf4f0 already in progress: creating tablet
I20250112 02:52:26.102954 1172 ts_tablet_manager.cc:1939] T 325e43fdf2fc4eacb48a5227e2ff3bf7 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
W20250112 02:52:26.105109 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet cb724feccc7f4fbc846fb6fa61be3385 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb724feccc7f4fbc846fb6fa61be3385 already in progress: creating tablet
W20250112 02:52:26.105396 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 161fe26b9c0e4543b7aeb0021506d72a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 161fe26b9c0e4543b7aeb0021506d72a already in progress: creating tablet
W20250112 02:52:26.111663 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3ee5d8a17874d7ba54de51568e7d2e6 consensus state has no leader
I20250112 02:52:26.113348 1172 tablet_replica.cc:331] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:26.113597 1172 raft_consensus.cc:2238] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:26.113881 1172 raft_consensus.cc:2267] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.115417 1172 ts_tablet_manager.cc:1905] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.116319 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_09e0faa7fd4b4531bce0741e3a2fa748"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.116963 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09e0faa7fd4b4531bce0741e3a2fa748 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.118605 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_c9cd1883fedc45e89202fd92fbbd0e50" } modify_external_catalogs: true
I20250112 02:52:26.118790 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_c9cd1883fedc45e89202fd92fbbd0e50" } modify_external_catalogs: true
I20250112 02:52:26.120011 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_3888b008030c447d885c3961d470cee2" } new_table_name: "default.Table_94d3a2be9a344af082b69ecfa503d4c0" modify_external_catalogs: true
I20250112 02:52:26.124552 892 tablet_service.cc:1467] Processing CreateTablet for tablet 89b70dce4d1d4e29997a18297ac0f82a (DEFAULT_TABLE table=default.Table_f9db9e236376446f8210cc4c9708ec6a [id=2fa03135646d4939b6efc7811ee4df10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.124969 895 tablet_service.cc:1467] Processing CreateTablet for tablet fccff64af8ca4400afc116db98918119 (DEFAULT_TABLE table=default.Table_3fec5f647ce141459d216dca71e5eb87 [id=ca9c8281bf254f248076cb600e7c4f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.124948 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89b70dce4d1d4e29997a18297ac0f82a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.130360 758 tablet_service.cc:1467] Processing CreateTablet for tablet 89b70dce4d1d4e29997a18297ac0f82a (DEFAULT_TABLE table=default.Table_f9db9e236376446f8210cc4c9708ec6a [id=2fa03135646d4939b6efc7811ee4df10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.130872 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89b70dce4d1d4e29997a18297ac0f82a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.131624 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fccff64af8ca4400afc116db98918119. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.135848 753 tablet_service.cc:1467] Processing CreateTablet for tablet fccff64af8ca4400afc116db98918119 (DEFAULT_TABLE table=default.Table_3fec5f647ce141459d216dca71e5eb87 [id=ca9c8281bf254f248076cb600e7c4f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.136138 898 tablet_service.cc:1467] Processing CreateTablet for tablet 9d5b3df88a0c45dba879b3d506710614 (DEFAULT_TABLE table=default.Table_49b5979e0b174043b61f2f1d61a5b58c [id=b81cf271b11e45dcb0b6bce102cd7a0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.136227 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fccff64af8ca4400afc116db98918119. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.136466 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d5b3df88a0c45dba879b3d506710614. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.139267 890 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1e3af2883341e9b957f6ed748e604d (DEFAULT_TABLE table=default.Table_b3ed6c7a79ed43568b8743f8aea9f67e [id=902e40c3f1e9401e934a4a2652f3dc22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.139595 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a1e3af2883341e9b957f6ed748e604d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.139786 751 tablet_service.cc:1467] Processing CreateTablet for tablet 9d5b3df88a0c45dba879b3d506710614 (DEFAULT_TABLE table=default.Table_49b5979e0b174043b61f2f1d61a5b58c [id=b81cf271b11e45dcb0b6bce102cd7a0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.140139 897 tablet_service.cc:1467] Processing CreateTablet for tablet 73136be25c79491784d3ea71aa9b27f2 (DEFAULT_TABLE table=default.Table_b8256597ba354774ace25add22300525 [id=d5bf19dc7bff469ba06018568b796a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.140206 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d5b3df88a0c45dba879b3d506710614. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.140461 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73136be25c79491784d3ea71aa9b27f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.143776 756 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1e3af2883341e9b957f6ed748e604d (DEFAULT_TABLE table=default.Table_b3ed6c7a79ed43568b8743f8aea9f67e [id=902e40c3f1e9401e934a4a2652f3dc22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.144094 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a1e3af2883341e9b957f6ed748e604d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.148041 760 tablet_service.cc:1467] Processing CreateTablet for tablet 73136be25c79491784d3ea71aa9b27f2 (DEFAULT_TABLE table=default.Table_b8256597ba354774ace25add22300525 [id=d5bf19dc7bff469ba06018568b796a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.148443 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73136be25c79491784d3ea71aa9b27f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.164788 1294 raft_consensus.cc:491] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:26.164978 1294 raft_consensus.cc:513] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.165685 1294 leader_election.cc:290] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:26.166585 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328e307a4b35445f92cbfab24f2ada48" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:26.167449 714 leader_election.cc:343] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.168025 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_dd5ce8476cb84204b0351fd07cab2b00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.168609 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd5ce8476cb84204b0351fd07cab2b00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.170281 525 catalog_manager.cc:6071] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: default.Table_3b1c87a5169e49b68fdf3d5c662a27bc [id=ed3672fa460a4aa2ae1d3b643ade43e6] alter complete (version 2)
I20250112 02:52:26.171591 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 49650a419dfd49d5a1547770d3f2fe54 with delete_type TABLET_DATA_DELETED (replaced by tablet d8eb0efc5d7f4c72bdfc7531a444d42a) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.171586 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_3d820aeff9534cc98f5722887aed5300"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.172726 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d820aeff9534cc98f5722887aed5300 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.176045 1269 raft_consensus.cc:491] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:26.177050 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 9427951452104a7db765437493b21e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.177459 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 328e307a4b35445f92cbfab24f2ada48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.179253 1269 raft_consensus.cc:513] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.180418 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328e307a4b35445f92cbfab24f2ada48" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:26.180466 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328e307a4b35445f92cbfab24f2ada48" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:26.180619 780 raft_consensus.cc:2463] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:26.181135 1269 leader_election.cc:290] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:26.182044 855 leader_election.cc:304] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
W20250112 02:52:26.182269 512 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:42341, user_credentials={real_user=slave}} blocked reactor thread for 76618.9us
I20250112 02:52:26.183244 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 73136be25c79491784d3ea71aa9b27f2 (DEFAULT_TABLE table=default.Table_b8256597ba354774ace25add22300525 [id=d5bf19dc7bff469ba06018568b796a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.183681 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73136be25c79491784d3ea71aa9b27f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.189666 1172 ts_tablet_manager.cc:1918] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20250112 02:52:26.190814 854 leader_election.cc:343] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.190922 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328e307a4b35445f92cbfab24f2ada48" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:26.191809 1172 log.cc:1198] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/c2595782d66e4e4b890c94221b196daa
I20250112 02:52:26.191974 918 raft_consensus.cc:2463] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:26.192322 1172 ts_tablet_manager.cc:1939] T c2595782d66e4e4b890c94221b196daa P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:26.192724 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 89b70dce4d1d4e29997a18297ac0f82a (DEFAULT_TABLE table=default.Table_f9db9e236376446f8210cc4c9708ec6a [id=2fa03135646d4939b6efc7811ee4df10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.193003 713 leader_election.cc:304] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:26.193070 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89b70dce4d1d4e29997a18297ac0f82a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.193368 1294 raft_consensus.cc:2798] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:26.193454 1294 raft_consensus.cc:491] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:26.193539 1294 raft_consensus.cc:3054] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:26.194530 1172 tablet_replica.cc:331] T aa67bf25dc894d7ea96d247a22c14e03 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:26.195356 1172 raft_consensus.cc:2238] T aa67bf25dc894d7ea96d247a22c14e03 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:26.195554 1172 raft_consensus.cc:2267] T aa67bf25dc894d7ea96d247a22c14e03 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.195952 1294 raft_consensus.cc:513] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.196115 1172 ts_tablet_manager.cc:1905] T aa67bf25dc894d7ea96d247a22c14e03 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.196888 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328e307a4b35445f92cbfab24f2ada48" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:26.197456 1294 leader_election.cc:290] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:26.197762 714 leader_election.cc:343] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.199707 1171 raft_consensus.cc:2238] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:26.200366 1171 pending_rounds.cc:62] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:26.200439 1171 pending_rounds.cc:69] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319817363111936 op_type: NO_OP noop_request { }
I20250112 02:52:26.200549 1171 raft_consensus.cc:2883] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:26.200649 1171 raft_consensus.cc:2267] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.203835 1171 ts_tablet_manager.cc:1905] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.212249 1172 ts_tablet_manager.cc:1918] T aa67bf25dc894d7ea96d247a22c14e03 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:26.212390 1172 log.cc:1198] T aa67bf25dc894d7ea96d247a22c14e03 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/aa67bf25dc894d7ea96d247a22c14e03
I20250112 02:52:26.212810 1172 ts_tablet_manager.cc:1939] T aa67bf25dc894d7ea96d247a22c14e03 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:26.214547 1171 ts_tablet_manager.cc:1918] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:26.214756 1171 log.cc:1198] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/aa67bf25dc894d7ea96d247a22c14e03
I20250112 02:52:26.216069 1175 raft_consensus.cc:2238] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:26.216145 1171 ts_tablet_manager.cc:1939] T aa67bf25dc894d7ea96d247a22c14e03 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:26.216279 1175 raft_consensus.cc:2267] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.216823 1175 ts_tablet_manager.cc:1905] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.217505 1171 tablet_replica.cc:331] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:26.217689 1171 raft_consensus.cc:2238] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:26.217887 1171 pending_rounds.cc:62] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:26.217960 1171 pending_rounds.cc:69] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319817004126208 op_type: NO_OP noop_request { }
I20250112 02:52:26.218075 1171 raft_consensus.cc:2883] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:26.218178 1171 raft_consensus.cc:2267] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.218931 1171 ts_tablet_manager.cc:1905] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.220049 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet aa67bf25dc894d7ea96d247a22c14e03 (table default.Table_7e88ecc0aa5e43ffb84af9b8dc6da938 [id=2f9194b28f5044e2a85f9e18fa041746]) successfully deleted
W20250112 02:52:26.220359 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 161fe26b9c0e4543b7aeb0021506d72a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 161fe26b9c0e4543b7aeb0021506d72a already in progress: creating tablet
W20250112 02:52:26.220552 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 1726d61cc36a4deb83c5d604e8daf4f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1726d61cc36a4deb83c5d604e8daf4f0 already in progress: creating tablet
W20250112 02:52:26.220781 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet cb724feccc7f4fbc846fb6fa61be3385 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb724feccc7f4fbc846fb6fa61be3385 already in progress: creating tablet
I20250112 02:52:26.221941 1028 tablet_service.cc:1467] Processing CreateTablet for tablet fccff64af8ca4400afc116db98918119 (DEFAULT_TABLE table=default.Table_3fec5f647ce141459d216dca71e5eb87 [id=ca9c8281bf254f248076cb600e7c4f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.222313 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fccff64af8ca4400afc116db98918119. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.228291 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1e3af2883341e9b957f6ed748e604d (DEFAULT_TABLE table=default.Table_b3ed6c7a79ed43568b8743f8aea9f67e [id=902e40c3f1e9401e934a4a2652f3dc22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.228616 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a1e3af2883341e9b957f6ed748e604d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.232731 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 325e43fdf2fc4eacb48a5227e2ff3bf7 (table default.Table_f7f876c63e68490c805c0bfa3f0c25ea [id=5c4769bfcefd4fe0a0f3c2e315d50c32]) successfully deleted
W20250112 02:52:26.232961 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet df6ccd04880442d0990c6cb0d27956bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df6ccd04880442d0990c6cb0d27956bd already in progress: creating tablet
W20250112 02:52:26.233114 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 49bb6d20cfa0453e90e048825114525e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49bb6d20cfa0453e90e048825114525e already in progress: creating tablet
W20250112 02:52:26.233255 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet ea33e5c1a1fb46599c5836a36b130051 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea33e5c1a1fb46599c5836a36b130051 already in progress: creating tablet
W20250112 02:52:26.233381 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet e9fc9a4f1e8944abb76dad6c2ad53d35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9fc9a4f1e8944abb76dad6c2ad53d35 already in progress: creating tablet
I20250112 02:52:26.233606 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet c2595782d66e4e4b890c94221b196daa (table default.Table_96a35d4e94284a49ba1a88d7d0acd3c1 [id=8559c454b17b4a00a60689a6a9b19f7d]) successfully deleted
I20250112 02:52:26.233767 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet aa67bf25dc894d7ea96d247a22c14e03 (table default.Table_7e88ecc0aa5e43ffb84af9b8dc6da938 [id=2f9194b28f5044e2a85f9e18fa041746]) successfully deleted
I20250112 02:52:26.236459 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 9d5b3df88a0c45dba879b3d506710614 (DEFAULT_TABLE table=default.Table_49b5979e0b174043b61f2f1d61a5b58c [id=b81cf271b11e45dcb0b6bce102cd7a0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.236792 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d5b3df88a0c45dba879b3d506710614. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.238567 1181 tablet_bootstrap.cc:492] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:26.238720 1181 ts_tablet_manager.cc:1397] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.358s user 0.040s sys 0.074s
I20250112 02:52:26.238831 1269 raft_consensus.cc:2798] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:26.240538 1173 tablet_bootstrap.cc:492] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:26.240664 1173 ts_tablet_manager.cc:1397] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.262s user 0.035s sys 0.073s
I20250112 02:52:26.241084 1269 raft_consensus.cc:491] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:26.241216 1269 raft_consensus.cc:3054] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:26.244109 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bb217afc08304c0ea35dee3120347ba1 consensus state has no leader
I20250112 02:52:26.246305 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_060115627d5f4bd786853027f080f260" } new_table_name: "default.Table_b46fcc60960d4de99ac8b7832afcb4b9" modify_external_catalogs: true
I20250112 02:52:26.246656 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 1c639a2bcd4740e4be7b2b162004669a
I20250112 02:52:26.247042 520 catalog_manager.cc:6596] ReplaceTablet: tablet 1c639a2bcd4740e4be7b2b162004669a deleted and replaced by tablet 5d0c3191973a477e949fbf9dfcca963f
I20250112 02:52:26.247485 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 1a1e3af2883341e9b957f6ed748e604d
I20250112 02:52:26.247736 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 35f59617b34046568dd83bbb7cdc42d2
I20250112 02:52:26.248112 519 master_service.cc:946] ReplaceTablet: received request to replace tablet 35f59617b34046568dd83bbb7cdc42d2 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:26.248193 521 master_service.cc:946] ReplaceTablet: received request to replace tablet 64707ee432fb4e96a3bbf7252dfa1813 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:26.248674 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_abf2c7b9b8da40c6b9a8381b4e41bda6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.249300 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abf2c7b9b8da40c6b9a8381b4e41bda6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.250389 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 6913b0ac9ceb4dbc9cdf591f159134f7
I20250112 02:52:26.250738 523 catalog_manager.cc:6596] ReplaceTablet: tablet 6913b0ac9ceb4dbc9cdf591f159134f7 deleted and replaced by tablet 416fb6820cd54e118950c68d381f51b2
I20250112 02:52:26.250908 523 master_service.cc:946] ReplaceTablet: received request to replace tablet 71b5f80ed198491780cbe17e6cd3fa15 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:26.251775 526 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 5dd48d0e68014e1aa0c44620527baec9
I20250112 02:52:26.252779 760 tablet_service.cc:1514] Processing DeleteTablet for tablet aa67bf25dc894d7ea96d247a22c14e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.253024 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 3567820f0d4349799b681fdfff0ebad8 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a18132549374218893f35a4e4226b72) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.253214 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1e3af2883341e9b957f6ed748e604d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.253388 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f59617b34046568dd83bbb7cdc42d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.246335 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_4f66980b946046a4bd2531c4ab094352" } modify_external_catalogs: true
I20250112 02:52:26.253561 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_4f66980b946046a4bd2531c4ab094352" } modify_external_catalogs: true
I20250112 02:52:26.253608 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd48d0e68014e1aa0c44620527baec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.253628 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c639a2bcd4740e4be7b2b162004669a with delete_type TABLET_DATA_DELETED (replaced by tablet 5d0c3191973a477e949fbf9dfcca963f) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.253832 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 416fb6820cd54e118950c68d381f51b2) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.255313 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet f6803d0f553b4204b415b2734286bf4a
I20250112 02:52:26.258199 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_d462fac16720400f879bf69c1cdbcd6d" } new_table_name: "default.Table_a23515295db7425e92ddbf1dcb7ace36" modify_external_catalogs: true
I20250112 02:52:26.258386 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_f7bab317410d408aa363857a99be6e3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.258823 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7bab317410d408aa363857a99be6e3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.260124 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_75db5f75b2874392b44dd580e0e52de9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.260540 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75db5f75b2874392b44dd580e0e52de9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.260461 1173 raft_consensus.cc:357] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.260649 1173 raft_consensus.cc:383] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.260762 1173 raft_consensus.cc:738] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.260970 1173 consensus_queue.cc:260] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.262181 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_47effe293cfd4ebfa64ad81407d7fe93" } new_table_name: "default.Table_32f2fd919490410aacdc9ca9c5a51683" modify_external_catalogs: true
I20250112 02:52:26.261561 1173 ts_tablet_manager.cc:1428] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20250112 02:52:26.263371 1173 tablet_bootstrap.cc:492] T 09886cd08c714f24a96e4df02bc0e6c8 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:26.247376 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_be0af4187400424dbc5afa4a18fd0fb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:26.265782 1173 tablet_bootstrap.cc:654] T 09886cd08c714f24a96e4df02bc0e6c8 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:26.266026 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be0af4187400424dbc5afa4a18fd0fb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.267808 756 tablet_service.cc:1514] Processing DeleteTablet for tablet f6803d0f553b4204b415b2734286bf4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.268687 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328e307a4b35445f92cbfab24f2ada48" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:26.271827 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd48d0e68014e1aa0c44620527baec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.274398 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd48d0e68014e1aa0c44620527baec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.274564 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f59617b34046568dd83bbb7cdc42d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.274879 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 416fb6820cd54e118950c68d381f51b2) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.275138 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c639a2bcd4740e4be7b2b162004669a with delete_type TABLET_DATA_DELETED (replaced by tablet 5d0c3191973a477e949fbf9dfcca963f) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.275317 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet c2595782d66e4e4b890c94221b196daa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.275503 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 416fb6820cd54e118950c68d381f51b2) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.275657 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c639a2bcd4740e4be7b2b162004669a with delete_type TABLET_DATA_DELETED (replaced by tablet 5d0c3191973a477e949fbf9dfcca963f) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.274585 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet f6803d0f553b4204b415b2734286bf4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.277388 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f59617b34046568dd83bbb7cdc42d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.277709 893 tablet_service.cc:1514] Processing DeleteTablet for tablet f6803d0f553b4204b415b2734286bf4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.278126 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1e3af2883341e9b957f6ed748e604d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.281692 1269 raft_consensus.cc:513] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.282094 918 raft_consensus.cc:2388] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 289afc116aa74408aa5bff74e11c50d8 in current term 1: Already voted for candidate fe36951a83fb4e57b26e187edbcbfe3d in this term.
I20250112 02:52:26.282732 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328e307a4b35445f92cbfab24f2ada48" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:26.282912 713 leader_election.cc:304] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
W20250112 02:52:26.283772 854 leader_election.cc:343] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.284731 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "328e307a4b35445f92cbfab24f2ada48" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:26.285022 780 raft_consensus.cc:2388] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fe36951a83fb4e57b26e187edbcbfe3d in current term 1: Already voted for candidate 289afc116aa74408aa5bff74e11c50d8 in this term.
I20250112 02:52:26.285511 855 leader_election.cc:304] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:26.285825 1190 raft_consensus.cc:2743] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250112 02:52:26.285919 1269 leader_election.cc:290] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:26.286310 1350 raft_consensus.cc:2743] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250112 02:52:26.287333 1175 ts_tablet_manager.cc:1918] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:26.287492 1175 log.cc:1198] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/71b5f80ed198491780cbe17e6cd3fa15
I20250112 02:52:26.287930 1175 ts_tablet_manager.cc:1939] T 71b5f80ed198491780cbe17e6cd3fa15 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:26.288985 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 71b5f80ed198491780cbe17e6cd3fa15 (table default.Table_247a821819994de6adac6fabd06c3a1f [id=60486b9e81254d74b8e08a1cadde13bd]) successfully deleted
I20250112 02:52:26.289551 1175 tablet_replica.cc:331] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:26.289901 1175 raft_consensus.cc:2238] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:26.290169 1175 raft_consensus.cc:2267] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.290534 424 master_runner.cc:186] Error getting master registration for 127.0.104.60:39987: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39987: connect: Connection refused (error 111),
I20250112 02:52:26.290737 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:52:26.290858 1175 ts_tablet_manager.cc:1905] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:26.294973 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 5dd48d0e68014e1aa0c44620527baec9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5dd48d0e68014e1aa0c44620527baec9 already in progress: creating tablet
W20250112 02:52:26.295588 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 35f59617b34046568dd83bbb7cdc42d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35f59617b34046568dd83bbb7cdc42d2 already in progress: creating tablet
W20250112 02:52:26.295840 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6913b0ac9ceb4dbc9cdf591f159134f7 already in progress: creating tablet
W20250112 02:52:26.296033 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 1c639a2bcd4740e4be7b2b162004669a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c639a2bcd4740e4be7b2b162004669a already in progress: creating tablet
W20250112 02:52:26.296257 512 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet c2595782d66e4e4b890c94221b196daa because the tablet was not found. No further retry: Not found: Tablet not found: c2595782d66e4e4b890c94221b196daa
W20250112 02:52:26.296442 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f6803d0f553b4204b415b2734286bf4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6803d0f553b4204b415b2734286bf4a already in progress: creating tablet
I20250112 02:52:26.299715 1302 raft_consensus.cc:491] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:26.299811 1181 raft_consensus.cc:357] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.299916 1302 raft_consensus.cc:513] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.300103 1181 raft_consensus.cc:383] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.300204 1181 raft_consensus.cc:738] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.300500 1181 consensus_queue.cc:260] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.300771 1302 leader_election.cc:290] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
W20250112 02:52:26.302100 989 leader_election.cc:343] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 28abc02b1b21427eae06560b99bcc078
I20250112 02:52:26.307166 1181 ts_tablet_manager.cc:1428] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.068s user 0.001s sys 0.001s
I20250112 02:52:26.307524 1181 tablet_bootstrap.cc:492] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
W20250112 02:52:26.308387 991 leader_election.cc:343] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: 28abc02b1b21427eae06560b99bcc078
I20250112 02:52:26.308532 991 leader_election.cc:304] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:26.308991 1302 raft_consensus.cc:2743] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:26.312089 1181 tablet_bootstrap.cc:654] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.313621 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bc067e895cb4c69af4dedc03533a276 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.313895 1172 tablet_replica.cc:331] T 4bc067e895cb4c69af4dedc03533a276 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:26.314126 1172 raft_consensus.cc:2238] T 4bc067e895cb4c69af4dedc03533a276 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:26.314301 1172 raft_consensus.cc:2267] T 4bc067e895cb4c69af4dedc03533a276 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.314951 1172 ts_tablet_manager.cc:1905] T 4bc067e895cb4c69af4dedc03533a276 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.322539 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1e3af2883341e9b957f6ed748e604d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.322875 1177 tablet_bootstrap.cc:492] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:26.323043 1177 ts_tablet_manager.cc:1397] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.298s user 0.026s sys 0.055s
I20250112 02:52:26.323827 1181 tablet_bootstrap.cc:492] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:26.323998 1181 ts_tablet_manager.cc:1397] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250112 02:52:26.324668 1181 raft_consensus.cc:357] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.324910 1181 raft_consensus.cc:383] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.325021 1181 raft_consensus.cc:738] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.325016 525 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b6dc79f4d1194f81963a8094ed6b0d2e (table default.Table_d462fac16720400f879bf69c1cdbcd6d [id=824805353d914188ba250e8624065190]). Expected version 1 got 0
I20250112 02:52:26.325277 1181 consensus_queue.cc:260] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.325896 1181 ts_tablet_manager.cc:1428] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:26.326705 1181 tablet_bootstrap.cc:492] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:26.328519 1181 tablet_bootstrap.cc:654] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.330404 1172 ts_tablet_manager.cc:1918] T 4bc067e895cb4c69af4dedc03533a276 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:26.330582 1172 log.cc:1198] T 4bc067e895cb4c69af4dedc03533a276 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/4bc067e895cb4c69af4dedc03533a276
I20250112 02:52:26.331038 1172 ts_tablet_manager.cc:1939] T 4bc067e895cb4c69af4dedc03533a276 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:26.331892 1177 raft_consensus.cc:357] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.332119 1177 raft_consensus.cc:383] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.332211 1177 raft_consensus.cc:738] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.332522 1177 consensus_queue.cc:260] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.333895 1177 ts_tablet_manager.cc:1428] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250112 02:52:26.334169 1177 tablet_bootstrap.cc:492] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:26.336359 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 28abc02b1b21427eae06560b99bcc078 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.336871 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 4bc067e895cb4c69af4dedc03533a276 (table default.Table_6e85e49c707f4021b9a16ca854bb2634 [id=03ca66b32af3425892725946ed90c493]) successfully deleted
W20250112 02:52:26.337153 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 1a1e3af2883341e9b957f6ed748e604d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a1e3af2883341e9b957f6ed748e604d already in progress: creating tablet
I20250112 02:52:26.337437 1172 tablet_replica.cc:331] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:26.337656 1172 raft_consensus.cc:2238] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:26.337846 1172 raft_consensus.cc:2267] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.338614 1172 ts_tablet_manager.cc:1905] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.342356 1171 ts_tablet_manager.cc:1918] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:26.342486 1171 log.cc:1198] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/df6ccd04880442d0990c6cb0d27956bd
I20250112 02:52:26.342877 1171 ts_tablet_manager.cc:1939] T df6ccd04880442d0990c6cb0d27956bd P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:26.349570 1173 tablet_bootstrap.cc:492] T 09886cd08c714f24a96e4df02bc0e6c8 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:26.349804 1173 ts_tablet_manager.cc:1397] T 09886cd08c714f24a96e4df02bc0e6c8 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.002s
I20250112 02:52:26.350244 1177 tablet_bootstrap.cc:654] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.350823 1173 raft_consensus.cc:357] T 09886cd08c714f24a96e4df02bc0e6c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.351115 1173 raft_consensus.cc:383] T 09886cd08c714f24a96e4df02bc0e6c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.351241 1173 raft_consensus.cc:738] T 09886cd08c714f24a96e4df02bc0e6c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.351635 1173 consensus_queue.cc:260] T 09886cd08c714f24a96e4df02bc0e6c8 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.352277 1173 ts_tablet_manager.cc:1428] T 09886cd08c714f24a96e4df02bc0e6c8 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:26.353638 1173 tablet_bootstrap.cc:492] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:26.355247 1173 tablet_bootstrap.cc:654] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.358361 1175 ts_tablet_manager.cc:1918] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:26.358611 1181 tablet_bootstrap.cc:492] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:26.358793 1181 ts_tablet_manager.cc:1397] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.019s
I20250112 02:52:26.359071 1175 log.cc:1198] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/d3e41e307df64109a41b0c1c44d3bda6
I20250112 02:52:26.359797 1181 raft_consensus.cc:357] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.360253 1181 raft_consensus.cc:383] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.360388 1181 raft_consensus.cc:738] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.360644 1181 consensus_queue.cc:260] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.361084 1181 ts_tablet_manager.cc:1428] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:26.361454 1181 tablet_bootstrap.cc:492] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
W20250112 02:52:26.362677 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b2ccb88108f24e36ba4b7ebf36398950 consensus state has no leader
I20250112 02:52:26.362982 1181 tablet_bootstrap.cc:654] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.364033 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_46a03fa2d75e4541a98a1b3d91f85395"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.364609 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46a03fa2d75e4541a98a1b3d91f85395 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.367735 1175 ts_tablet_manager.cc:1939] T d3e41e307df64109a41b0c1c44d3bda6 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:26.368851 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet d3e41e307df64109a41b0c1c44d3bda6 (table default.Table_01a0b2e48fcd4a769a3d442390622c1e [id=4d04607bb9004f3cb1871c43d0eb47ce]) successfully deleted
I20250112 02:52:26.370718 1181 tablet_bootstrap.cc:492] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:26.370869 1181 ts_tablet_manager.cc:1397] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20250112 02:52:26.371433 1181 raft_consensus.cc:357] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.371706 1181 raft_consensus.cc:383] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.371798 1181 raft_consensus.cc:738] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.372628 1181 consensus_queue.cc:260] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.375614 1181 ts_tablet_manager.cc:1428] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250112 02:52:26.377096 1181 tablet_bootstrap.cc:492] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:26.378070 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet df6ccd04880442d0990c6cb0d27956bd (table default.Table_c667abf579554ea0af764de6515a18c6 [id=a8c0e9da5acc45c3ab7f67427f59e2b6]) successfully deleted
W20250112 02:52:26.378705 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 49bb6d20cfa0453e90e048825114525e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49bb6d20cfa0453e90e048825114525e already in progress: creating tablet
W20250112 02:52:26.379252 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet ea33e5c1a1fb46599c5836a36b130051 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea33e5c1a1fb46599c5836a36b130051 already in progress: creating tablet
I20250112 02:52:26.379597 1171 tablet_replica.cc:331] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:26.379855 1171 raft_consensus.cc:2238] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:26.380121 1171 pending_rounds.cc:62] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:26.380350 1171 pending_rounds.cc:69] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319816638681088 op_type: NO_OP noop_request { }
I20250112 02:52:26.380537 1171 raft_consensus.cc:2883] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:26.380705 1171 raft_consensus.cc:2267] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.380714 1172 ts_tablet_manager.cc:1918] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:26.380746 1181 tablet_bootstrap.cc:654] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.380821 1172 log.cc:1198] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/28abc02b1b21427eae06560b99bcc078
I20250112 02:52:26.381160 1172 ts_tablet_manager.cc:1939] T 28abc02b1b21427eae06560b99bcc078 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:26.382073 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 28abc02b1b21427eae06560b99bcc078 (table default.Table_2bceb576abd94da4a7aca29c4b4fa89a [id=703dedbc15f549298f82cbbb62ee27d8]) successfully deleted
I20250112 02:52:26.382355 1175 tablet_replica.cc:331] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:26.382550 1175 raft_consensus.cc:2238] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:26.386075 1177 tablet_bootstrap.cc:492] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:26.386231 1177 ts_tablet_manager.cc:1397] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.000s
W20250112 02:52:26.387185 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6dc79f4d1194f81963a8094ed6b0d2e consensus state has no leader
I20250112 02:52:26.387615 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_f22fde4d1df34a7b955de7c6eee20c3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.388173 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f22fde4d1df34a7b955de7c6eee20c3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.388581 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_ea7aa744f34e4d5ebcffb74ae74533d8" } new_table_name: "default.Table_9b65dd291c9f4fcdaa6d5e37f04d24ae" modify_external_catalogs: true
W20250112 02:52:26.390179 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5cc414d6420c41839621ab858067548b has no consensus state
I20250112 02:52:26.390532 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_525750dad2b7461586d070dbaa74d0b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.390987 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_525750dad2b7461586d070dbaa74d0b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.391296 1177 raft_consensus.cc:357] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.391525 1177 raft_consensus.cc:383] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.391593 1177 raft_consensus.cc:738] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.391845 1177 consensus_queue.cc:260] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.392052 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 64707ee432fb4e96a3bbf7252dfa1813
I20250112 02:52:26.392480 521 catalog_manager.cc:6596] ReplaceTablet: tablet 64707ee432fb4e96a3bbf7252dfa1813 deleted and replaced by tablet b3c5a861a90840d0a33b780ec75bffd2
I20250112 02:52:26.393838 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 64707ee432fb4e96a3bbf7252dfa1813 with delete_type TABLET_DATA_DELETED (replaced by tablet b3c5a861a90840d0a33b780ec75bffd2) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.400038 1177 ts_tablet_manager.cc:1428] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250112 02:52:26.403306 1177 tablet_bootstrap.cc:492] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:26.403666 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 09886cd08c714f24a96e4df02bc0e6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.404562 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 64707ee432fb4e96a3bbf7252dfa1813 with delete_type TABLET_DATA_DELETED (replaced by tablet b3c5a861a90840d0a33b780ec75bffd2) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.405196 1177 tablet_bootstrap.cc:654] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.408394 1171 ts_tablet_manager.cc:1905] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.413189 492 master_runner.cc:186] Error getting master registration for 127.0.104.60:39987: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39987: connect: Connection refused (error 111),
I20250112 02:52:26.413367 492 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:52:26.388691 1181 tablet_bootstrap.cc:492] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:26.393474 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 64707ee432fb4e96a3bbf7252dfa1813 with delete_type TABLET_DATA_DELETED (replaced by tablet b3c5a861a90840d0a33b780ec75bffd2) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.414161 1181 ts_tablet_manager.cc:1397] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.004s
I20250112 02:52:26.414270 1175 raft_consensus.cc:2267] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.415045 1175 ts_tablet_manager.cc:1905] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.416086 1181 raft_consensus.cc:357] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.416370 1181 raft_consensus.cc:383] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.416478 1181 raft_consensus.cc:738] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.416816 1181 consensus_queue.cc:260] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.417723 1181 ts_tablet_manager.cc:1428] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250112 02:52:26.418706 1181 tablet_bootstrap.cc:492] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:26.419493 1171 ts_tablet_manager.cc:1918] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:26.421525 1173 tablet_bootstrap.cc:492] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:26.421659 1173 ts_tablet_manager.cc:1397] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.068s user 0.006s sys 0.024s
I20250112 02:52:26.422498 1173 raft_consensus.cc:357] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.422683 1173 raft_consensus.cc:383] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.422789 1173 raft_consensus.cc:738] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.423089 1173 consensus_queue.cc:260] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.424702 1171 log.cc:1198] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/4bc067e895cb4c69af4dedc03533a276
I20250112 02:52:26.425361 1171 ts_tablet_manager.cc:1939] T 4bc067e895cb4c69af4dedc03533a276 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:26.425364 1173 ts_tablet_manager.cc:1428] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 02:52:26.425642 1173 tablet_bootstrap.cc:492] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:26.426194 1302 raft_consensus.cc:491] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:26.426342 1302 raft_consensus.cc:513] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.426934 1302 leader_election.cc:290] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:26.427254 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 4bc067e895cb4c69af4dedc03533a276 (table default.Table_6e85e49c707f4021b9a16ca854bb2634 [id=03ca66b32af3425892725946ed90c493]) successfully deleted
W20250112 02:52:26.427883 991 leader_election.cc:343] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: f05d3326545849b9af4519bc66c21069
W20250112 02:52:26.428274 989 leader_election.cc:343] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: f05d3326545849b9af4519bc66c21069
I20250112 02:52:26.428426 989 leader_election.cc:304] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:26.428742 1181 tablet_bootstrap.cc:654] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.428952 1302 raft_consensus.cc:2743] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:26.432556 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet e9fc9a4f1e8944abb76dad6c2ad53d35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9fc9a4f1e8944abb76dad6c2ad53d35 already in progress: creating tablet
I20250112 02:52:26.435806 1177 tablet_bootstrap.cc:492] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:26.435966 1177 ts_tablet_manager.cc:1397] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
W20250112 02:52:26.437880 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7cd9985cd20148f195a857c478b0755f consensus state has no leader
I20250112 02:52:26.439774 1177 raft_consensus.cc:357] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.440007 1177 raft_consensus.cc:383] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.440095 1177 raft_consensus.cc:738] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.440330 1177 consensus_queue.cc:260] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.441988 1177 ts_tablet_manager.cc:1428] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250112 02:52:26.442330 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 6117dc8b32a64535a2f4167a03bed0d2 (DEFAULT_TABLE table=default.Table_c734b496a0634288b6a93d8007262510 [id=868adb627654450c936ef198e49d853b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.442732 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6117dc8b32a64535a2f4167a03bed0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.444319 756 tablet_service.cc:1467] Processing CreateTablet for tablet 6117dc8b32a64535a2f4167a03bed0d2 (DEFAULT_TABLE table=default.Table_c734b496a0634288b6a93d8007262510 [id=868adb627654450c936ef198e49d853b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.444681 760 tablet_service.cc:1467] Processing CreateTablet for tablet 352c87f1dc724e7b94bc2e64abfce8d7 (DEFAULT_TABLE table=default.Table_2e35ffa5571d416d9b68443869009ab4 [id=399c12e9075d4e55b21b17837ead62da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.445058 751 tablet_service.cc:1467] Processing CreateTablet for tablet 1b184806b4884cd3b0a002905a1e57af (DEFAULT_TABLE table=default.Table_bac47d0dd2ab473babf687f5df1691c0 [id=78ee5f3a07714849a10192e80c425264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.446754 1171 tablet_replica.cc:331] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:26.447186 888 tablet_service.cc:1467] Processing CreateTablet for tablet f7cd0927c6a4401cbe69f55d4db9963a (DEFAULT_TABLE table=default.Table_8b4baeb21bec40af9997451f3eceddf1 [id=b7a72bcbde114d5992782721007a0cd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.447665 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7cd0927c6a4401cbe69f55d4db9963a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.449170 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6117dc8b32a64535a2f4167a03bed0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.451164 895 tablet_service.cc:1467] Processing CreateTablet for tablet 6117dc8b32a64535a2f4167a03bed0d2 (DEFAULT_TABLE table=default.Table_c734b496a0634288b6a93d8007262510 [id=868adb627654450c936ef198e49d853b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.451499 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6117dc8b32a64535a2f4167a03bed0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.451954 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 352c87f1dc724e7b94bc2e64abfce8d7 (DEFAULT_TABLE table=default.Table_2e35ffa5571d416d9b68443869009ab4 [id=399c12e9075d4e55b21b17837ead62da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.452394 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352c87f1dc724e7b94bc2e64abfce8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.452389 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352c87f1dc724e7b94bc2e64abfce8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.455672 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b184806b4884cd3b0a002905a1e57af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.455880 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 1b184806b4884cd3b0a002905a1e57af (DEFAULT_TABLE table=default.Table_bac47d0dd2ab473babf687f5df1691c0 [id=78ee5f3a07714849a10192e80c425264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.456184 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b184806b4884cd3b0a002905a1e57af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.456315 887 tablet_service.cc:1467] Processing CreateTablet for tablet 8ee98f055024429eafaa29bb961cab87 (DEFAULT_TABLE table=default.Table_0c0859249ddd40ea8224fa9b119b3af3 [id=d1fa33e075d74f1193fe708f5034fbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.456681 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ee98f055024429eafaa29bb961cab87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.457464 886 tablet_service.cc:1467] Processing CreateTablet for tablet f31b777e7e0f43a28839d8c48c6a6590 (DEFAULT_TABLE table=default.Table_45c4d5f155af43818f8fe32452dcf5d8 [id=6402abce8d1d45c182cc989490c6a5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.457738 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31b777e7e0f43a28839d8c48c6a6590. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.458112 890 tablet_service.cc:1467] Processing CreateTablet for tablet 1811ce8982fd4003a35e89ea34e32746 (DEFAULT_TABLE table=default.Table_3e58cdf6ca254711b8071a2899d2402f [id=6a788a11657f4098bc906e16cd54618e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.458565 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1811ce8982fd4003a35e89ea34e32746. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.459756 897 tablet_service.cc:1467] Processing CreateTablet for tablet 1dda52f012794493a102ed1abc718f5b (DEFAULT_TABLE table=default.Table_ab4d7c58924a4b8682b52637c23f125b [id=a622f7a973344a388b3275adbcce6234]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.460129 893 tablet_service.cc:1467] Processing CreateTablet for tablet 352c87f1dc724e7b94bc2e64abfce8d7 (DEFAULT_TABLE table=default.Table_2e35ffa5571d416d9b68443869009ab4 [id=399c12e9075d4e55b21b17837ead62da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.460376 892 tablet_service.cc:1467] Processing CreateTablet for tablet 2730634a5cfd4c2fb5c4e30100c556d4 (DEFAULT_TABLE table=default.Table_548027a9e05c4376b024df028f14e86d [id=274b4ed84aa7490dbb78270b48150b73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.460554 898 tablet_service.cc:1467] Processing CreateTablet for tablet 1b184806b4884cd3b0a002905a1e57af (DEFAULT_TABLE table=default.Table_bac47d0dd2ab473babf687f5df1691c0 [id=78ee5f3a07714849a10192e80c425264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.465082 525 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c69587c5a09d40f0baba5c2f6e8a7b41 (table default.Table_1216acc1fa6b4e51a55a89ffe22bcc9a [id=b5457e4343ee43618c8309e737b69ec3]). Expected version 1 got 0
I20250112 02:52:26.465226 753 tablet_service.cc:1467] Processing CreateTablet for tablet 2730634a5cfd4c2fb5c4e30100c556d4 (DEFAULT_TABLE table=default.Table_548027a9e05c4376b024df028f14e86d [id=274b4ed84aa7490dbb78270b48150b73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.465517 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2730634a5cfd4c2fb5c4e30100c556d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.467839 758 tablet_service.cc:1467] Processing CreateTablet for tablet 8ee98f055024429eafaa29bb961cab87 (DEFAULT_TABLE table=default.Table_0c0859249ddd40ea8224fa9b119b3af3 [id=d1fa33e075d74f1193fe708f5034fbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.468222 754 tablet_service.cc:1467] Processing CreateTablet for tablet 1dda52f012794493a102ed1abc718f5b (DEFAULT_TABLE table=default.Table_ab4d7c58924a4b8682b52637c23f125b [id=a622f7a973344a388b3275adbcce6234]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.468561 749 tablet_service.cc:1467] Processing CreateTablet for tablet f31b777e7e0f43a28839d8c48c6a6590 (DEFAULT_TABLE table=default.Table_45c4d5f155af43818f8fe32452dcf5d8 [id=6402abce8d1d45c182cc989490c6a5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.468883 750 tablet_service.cc:1467] Processing CreateTablet for tablet f7cd0927c6a4401cbe69f55d4db9963a (DEFAULT_TABLE table=default.Table_8b4baeb21bec40af9997451f3eceddf1 [id=b7a72bcbde114d5992782721007a0cd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.469218 757 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc414d6420c41839621ab858067548b (DEFAULT_TABLE table=default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.468227 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ee98f055024429eafaa29bb961cab87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.474126 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dda52f012794493a102ed1abc718f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.476485 760 tablet_service.cc:1467] Processing CreateTablet for tablet ac716c6da01941a989754aac1ef430d5 (DEFAULT_TABLE table=default.Table_bee5a719ea4b44b78254bae566a0ba92 [id=d3fa1ea0552e4aaeabd6ac3ac84ca389]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.476830 756 tablet_service.cc:1467] Processing CreateTablet for tablet 1811ce8982fd4003a35e89ea34e32746 (DEFAULT_TABLE table=default.Table_3e58cdf6ca254711b8071a2899d2402f [id=6a788a11657f4098bc906e16cd54618e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.477155 1173 tablet_bootstrap.cc:654] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.479982 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31b777e7e0f43a28839d8c48c6a6590. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.483613 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7cd0927c6a4401cbe69f55d4db9963a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.484645 1177 tablet_bootstrap.cc:492] T d3e41e307df64109a41b0c1c44d3bda6 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:26.484959 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dda52f012794493a102ed1abc718f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.484928 891 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc414d6420c41839621ab858067548b (DEFAULT_TABLE table=default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.485707 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cc414d6420c41839621ab858067548b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.487239 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b184806b4884cd3b0a002905a1e57af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.487656 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cc414d6420c41839621ab858067548b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.489256 885 tablet_service.cc:1514] Processing DeleteTablet for tablet a47cdc208fbc431b8326364e84432cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.489482 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2730634a5cfd4c2fb5c4e30100c556d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.490561 1177 tablet_bootstrap.cc:654] T d3e41e307df64109a41b0c1c44d3bda6 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.491418 896 tablet_service.cc:1467] Processing CreateTablet for tablet ac716c6da01941a989754aac1ef430d5 (DEFAULT_TABLE table=default.Table_bee5a719ea4b44b78254bae566a0ba92 [id=d3fa1ea0552e4aaeabd6ac3ac84ca389]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.491999 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 99715f37fbef47c6860fa1d95e350f20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.492431 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352c87f1dc724e7b94bc2e64abfce8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.493266 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac716c6da01941a989754aac1ef430d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.493517 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_8b4baeb21bec40af9997451f3eceddf1" } modify_external_catalogs: true
I20250112 02:52:26.493742 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_8b4baeb21bec40af9997451f3eceddf1" } modify_external_catalogs: true
I20250112 02:52:26.493816 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_c0aaadb5e69e46e8983f4d289a44afe2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.494318 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0aaadb5e69e46e8983f4d289a44afe2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.494983 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_7e09abe4b91b4b2db2bf7d3085e19502"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.495579 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e09abe4b91b4b2db2bf7d3085e19502 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.496064 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_be0af4187400424dbc5afa4a18fd0fb1" } new_table_name: "default.Table_ce0ccdae99624243bdac1499da1790ee" modify_external_catalogs: true
I20250112 02:52:26.496747 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_3fcca85a33054d7ab388f29bba85fe35" } modify_external_catalogs: true
I20250112 02:52:26.496912 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_3fcca85a33054d7ab388f29bba85fe35" } modify_external_catalogs: true
I20250112 02:52:26.497216 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_0188036e1bd0455ea82fd355934295fc" } modify_external_catalogs: true
I20250112 02:52:26.497426 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_0188036e1bd0455ea82fd355934295fc" } modify_external_catalogs: true
I20250112 02:52:26.493573 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_49b5979e0b174043b61f2f1d61a5b58c" } modify_external_catalogs: true
I20250112 02:52:26.498298 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_49b5979e0b174043b61f2f1d61a5b58c" } modify_external_catalogs: true
I20250112 02:52:26.495441 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_16b036d8833a47a99577ab67579ccb5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.499909 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16b036d8833a47a99577ab67579ccb5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.512089 1173 tablet_bootstrap.cc:492] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:26.521309 1245 raft_consensus.cc:491] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:26.536300 1171 raft_consensus.cc:2238] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:26.537137 1245 raft_consensus.cc:513] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.537443 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac716c6da01941a989754aac1ef430d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.538966 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1811ce8982fd4003a35e89ea34e32746. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.539976 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9533ea72f6420bb55a53cdfc9125f7" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:26.540406 918 raft_consensus.cc:2463] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:26.541174 1245 leader_election.cc:290] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:26.541364 713 leader_election.cc:304] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:26.541682 1245 raft_consensus.cc:2798] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:26.541848 1171 raft_consensus.cc:2267] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.544590 1245 raft_consensus.cc:491] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:26.544906 1245 raft_consensus.cc:3054] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:26.548132 1171 ts_tablet_manager.cc:1905] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.548452 1245 raft_consensus.cc:513] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.549625 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9533ea72f6420bb55a53cdfc9125f7" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:26.549849 918 raft_consensus.cc:3054] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:26.552388 918 raft_consensus.cc:2463] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:26.552953 713 leader_election.cc:304] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:26.553194 1294 raft_consensus.cc:2798] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:26.554124 1173 ts_tablet_manager.cc:1397] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.129s user 0.004s sys 0.000s
I20250112 02:52:26.554195 1294 raft_consensus.cc:695] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:26.554454 1294 consensus_queue.cc:237] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.554898 1173 raft_consensus.cc:357] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.555119 1173 raft_consensus.cc:383] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.555198 1173 raft_consensus.cc:738] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.555389 1173 consensus_queue.cc:260] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.558332 1173 ts_tablet_manager.cc:1428] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:26.560382 1245 leader_election.cc:290] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:26.562361 1269 raft_consensus.cc:491] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:26.562595 1269 raft_consensus.cc:513] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.563401 1269 leader_election.cc:290] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:26.564651 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47cdc208fbc431b8326364e84432cf5" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:26.564980 1181 tablet_bootstrap.cc:492] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:26.565129 1181 ts_tablet_manager.cc:1397] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.146s user 0.006s sys 0.005s
W20250112 02:52:26.565187 854 leader_election.cc:343] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.565835 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 2730634a5cfd4c2fb5c4e30100c556d4 (DEFAULT_TABLE table=default.Table_548027a9e05c4376b024df028f14e86d [id=274b4ed84aa7490dbb78270b48150b73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.565827 1181 raft_consensus.cc:357] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.565918 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 8ee98f055024429eafaa29bb961cab87 (DEFAULT_TABLE table=default.Table_0c0859249ddd40ea8224fa9b119b3af3 [id=d1fa33e075d74f1193fe708f5034fbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.566017 1181 raft_consensus.cc:383] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.566083 1181 raft_consensus.cc:738] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.566290 1181 consensus_queue.cc:260] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.566540 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 1dda52f012794493a102ed1abc718f5b (DEFAULT_TABLE table=default.Table_ab4d7c58924a4b8682b52637c23f125b [id=a622f7a973344a388b3275adbcce6234]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.566545 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47cdc208fbc431b8326364e84432cf5" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:26.566761 1181 ts_tablet_manager.cc:1428] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:26.567066 1181 tablet_bootstrap.cc:492] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
W20250112 02:52:26.567343 855 leader_election.cc:343] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.567457 855 leader_election.cc:304] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:26.567682 1031 tablet_service.cc:1467] Processing CreateTablet for tablet f31b777e7e0f43a28839d8c48c6a6590 (DEFAULT_TABLE table=default.Table_45c4d5f155af43818f8fe32452dcf5d8 [id=6402abce8d1d45c182cc989490c6a5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.567962 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet f05d3326545849b9af4519bc66c21069 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.568164 1269 raft_consensus.cc:2743] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:26.568143 1029 tablet_service.cc:1467] Processing CreateTablet for tablet f7cd0927c6a4401cbe69f55d4db9963a (DEFAULT_TABLE table=default.Table_8b4baeb21bec40af9997451f3eceddf1 [id=b7a72bcbde114d5992782721007a0cd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.568622 1181 tablet_bootstrap.cc:654] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.568888 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9533ea72f6420bb55a53cdfc9125f7" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:26.569203 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc414d6420c41839621ab858067548b (DEFAULT_TABLE table=default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.569295 1034 tablet_service.cc:1467] Processing CreateTablet for tablet ac716c6da01941a989754aac1ef430d5 (DEFAULT_TABLE table=default.Table_bee5a719ea4b44b78254bae566a0ba92 [id=d3fa1ea0552e4aaeabd6ac3ac84ca389]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.569947 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9533ea72f6420bb55a53cdfc9125f7" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:26.555029 1172 tablet_replica.cc:331] T 09886cd08c714f24a96e4df02bc0e6c8 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
W20250112 02:52:26.573864 714 leader_election.cc:343] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.574071 1177 tablet_bootstrap.cc:492] T d3e41e307df64109a41b0c1c44d3bda6 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:26.574131 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ee98f055024429eafaa29bb961cab87. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:26.574456 714 leader_election.cc:343] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.576283 1172 raft_consensus.cc:2238] T 09886cd08c714f24a96e4df02bc0e6c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:26.581456 1177 ts_tablet_manager.cc:1397] T d3e41e307df64109a41b0c1c44d3bda6 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.090s user 0.006s sys 0.001s
I20250112 02:52:26.583510 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 54033a747d5a49b0a1c16f138158f596
I20250112 02:52:26.584453 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 54033a747d5a49b0a1c16f138158f596 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.586071 897 tablet_service.cc:1514] Processing DeleteTablet for tablet f9186ac46ed441ee9308490e90da0e23 with delete_type TABLET_DATA_DELETED (replaced by tablet 04f135b226774285a077e5806131d91b) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.586992 524 master_service.cc:946] ReplaceTablet: received request to replace tablet 8a12b225131b464494a9f02133d37aea from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:26.587705 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet f7cd0927c6a4401cbe69f55d4db9963a
I20250112 02:52:26.588150 518 master_service.cc:946] ReplaceTablet: received request to replace tablet fb0aba0139cf479b80b32f3dfc936535 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:26.589527 521 master_service.cc:946] ReplaceTablet: received request to replace tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:26.589977 521 master_service.cc:946] ReplaceTablet: received request to replace tablet fb0aba0139cf479b80b32f3dfc936535 from {username='slave'} at 127.0.0.1:40934
W20250112 02:52:26.590356 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8b64e4e64cc462891f1ee7a594d5026 has no consensus state
I20250112 02:52:26.590730 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_0f67e34fb9604f57bbf8656d531ce52c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.591223 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f67e34fb9604f57bbf8656d531ce52c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.592315 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 7a18132549374218893f35a4e4226b72
I20250112 02:52:26.592871 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_0c0859249ddd40ea8224fa9b119b3af3" } new_table_name: "default.Table_6e30b071de75435da0245b469fb16ff1" modify_external_catalogs: true
I20250112 02:52:26.594003 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 61593969d2234ce3a6089027b067d922
I20250112 02:52:26.594614 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 9d5b3df88a0c45dba879b3d506710614
I20250112 02:52:26.594370 1177 raft_consensus.cc:357] T d3e41e307df64109a41b0c1c44d3bda6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.594923 1177 raft_consensus.cc:383] T d3e41e307df64109a41b0c1c44d3bda6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.594997 1177 raft_consensus.cc:738] T d3e41e307df64109a41b0c1c44d3bda6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.595223 1177 consensus_queue.cc:260] T d3e41e307df64109a41b0c1c44d3bda6 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.596860 897 tablet_service.cc:1514] Processing DeleteTablet for tablet f7cd0927c6a4401cbe69f55d4db9963a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.596544 1181 tablet_bootstrap.cc:492] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:26.596927 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a18132549374218893f35a4e4226b72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.597245 1177 ts_tablet_manager.cc:1428] T d3e41e307df64109a41b0c1c44d3bda6 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20250112 02:52:26.597406 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 61593969d2234ce3a6089027b067d922 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.597559 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d5b3df88a0c45dba879b3d506710614 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.598018 1177 tablet_bootstrap.cc:492] T fb4c004397ee480d897e8b125d248829 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:26.598133 1181 ts_tablet_manager.cc:1397] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.004s
I20250112 02:52:26.602006 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_7a3faaceec4f4bd29b53455ec1228c80" } new_table_name: "default.Table_cd1f185d82b3474083246ec3702851fc" modify_external_catalogs: true
I20250112 02:52:26.604163 1181 raft_consensus.cc:357] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.606849 1171 ts_tablet_manager.cc:1918] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:26.606729 1181 raft_consensus.cc:383] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.607856 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_09e0faa7fd4b4531bce0741e3a2fa748" } new_table_name: "default.Table_b2b3c738a24f451eac39fb1a6399f37a" modify_external_catalogs: true
I20250112 02:52:26.607908 1181 raft_consensus.cc:738] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.608691 1181 consensus_queue.cc:260] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.610121 1181 ts_tablet_manager.cc:1428] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.011s user 0.000s sys 0.003s
I20250112 02:52:26.610396 1181 tablet_bootstrap.cc:492] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:26.609491 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d5b3df88a0c45dba879b3d506710614 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.611140 1177 tablet_bootstrap.cc:654] T fb4c004397ee480d897e8b125d248829 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.612051 1181 tablet_bootstrap.cc:654] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.612237 1171 log.cc:1198] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/c2595782d66e4e4b890c94221b196daa
I20250112 02:52:26.612735 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 97b31ebe035c4552baaffc3133ec73c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.613314 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_0e7ff0c4e04e4b23a8baf069f96ac465"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.613832 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e7ff0c4e04e4b23a8baf069f96ac465 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.614753 1171 ts_tablet_manager.cc:1939] T c2595782d66e4e4b890c94221b196daa P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:26.617468 1181 tablet_bootstrap.cc:492] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:26.617607 1181 ts_tablet_manager.cc:1397] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250112 02:52:26.576478 1172 raft_consensus.cc:2267] T 09886cd08c714f24a96e4df02bc0e6c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.607659 1302 raft_consensus.cc:491] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:26.618542 1302 raft_consensus.cc:513] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.618916 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 61593969d2234ce3a6089027b067d922 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.619050 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d5b3df88a0c45dba879b3d506710614 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.619141 1172 ts_tablet_manager.cc:1905] T 09886cd08c714f24a96e4df02bc0e6c8 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.619289 1302 leader_election.cc:290] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:26.581229 1173 tablet_bootstrap.cc:492] T 96e201d798c94e448a78e1c8adf41724 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:26.618247 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet f7cd0927c6a4401cbe69f55d4db9963a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.580127 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dda52f012794493a102ed1abc718f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.618551 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a18132549374218893f35a4e4226b72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.620131 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cc414d6420c41839621ab858067548b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.618031 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 54033a747d5a49b0a1c16f138158f596 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.622351 1173 tablet_bootstrap.cc:654] T 96e201d798c94e448a78e1c8adf41724 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.623587 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31b777e7e0f43a28839d8c48c6a6590. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:26.624691 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 93786us
I20250112 02:52:26.627656 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2730634a5cfd4c2fb5c4e30100c556d4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:26.628329 989 leader_election.cc:343] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 235e9dd0964c4ea6af7d05ef58acae0d
I20250112 02:52:26.629784 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 1811ce8982fd4003a35e89ea34e32746 (DEFAULT_TABLE table=default.Table_3e58cdf6ca254711b8071a2899d2402f [id=6a788a11657f4098bc906e16cd54618e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.630123 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1811ce8982fd4003a35e89ea34e32746. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:26.631085 991 leader_election.cc:343] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: 235e9dd0964c4ea6af7d05ef58acae0d
I20250112 02:52:26.631222 991 leader_election.cc:304] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:26.631630 1302 raft_consensus.cc:2743] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:26.632189 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac716c6da01941a989754aac1ef430d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.632864 1181 raft_consensus.cc:357] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.633133 1181 raft_consensus.cc:383] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.633217 1181 raft_consensus.cc:738] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.633478 1181 consensus_queue.cc:260] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.633975 1181 ts_tablet_manager.cc:1428] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20250112 02:52:26.634264 1181 tablet_bootstrap.cc:492] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:26.635567 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7cd0927c6a4401cbe69f55d4db9963a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.636611 1175 ts_tablet_manager.cc:1918] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:26.637820 1175 log.cc:1198] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/328e307a4b35445f92cbfab24f2ada48
I20250112 02:52:26.638355 1175 ts_tablet_manager.cc:1939] T 328e307a4b35445f92cbfab24f2ada48 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:26.638796 1269 raft_consensus.cc:491] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:26.639744 1181 tablet_bootstrap.cc:654] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.639802 1269 raft_consensus.cc:513] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.640432 1269 leader_election.cc:290] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:26.641767 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 187) took 1159 ms. Trace:
I20250112 02:52:26.642992 1269 raft_consensus.cc:491] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:26.643147 1269 raft_consensus.cc:513] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.643824 1269 leader_election.cc:290] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:26.644001 1229 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:39987
I20250112 02:52:26.644022 1439 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:39987 every 8 connection(s)
I20250112 02:52:26.642993 1175 rpcz_store.cc:276] 0112 02:52:25.482366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:25.484294 (+ 1928us) service_pool.cc:225] Handling call
0112 02:52:26.382281 (+897987us) ts_tablet_manager.cc:1145] Deleting tablet 328e307a4b35445f92cbfab24f2ada48
0112 02:52:26.382289 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:26.552730 (+170441us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:26.636488 (+ 83758us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:26.641729 (+ 5241us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":897675}
I20250112 02:52:26.645301 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 180) took 1163 ms. Trace:
I20250112 02:52:26.645325 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 328e307a4b35445f92cbfab24f2ada48 (table default.Table_fc5d684c7f2049e5b2976823cfb242ae [id=c84a7bdac0ed4c2ea7c0599d5f1ca9c1]) successfully deleted
I20250112 02:52:26.645484 1175 rpcz_store.cc:276] 0112 02:52:25.481512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:25.486068 (+ 4556us) service_pool.cc:225] Handling call
0112 02:52:26.645164 (+1159096us) ts_tablet_manager.cc:1145] Deleting tablet 161fe26b9c0e4543b7aeb0021506d72a
0112 02:52:26.645173 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:26.645265 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1158893}
W20250112 02:52:26.645895 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 161fe26b9c0e4543b7aeb0021506d72a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 161fe26b9c0e4543b7aeb0021506d72a already in progress: creating tablet
I20250112 02:52:26.645977 1229 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/data/info.pb
I20250112 02:52:26.647336 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 188) took 1141 ms. Trace:
I20250112 02:52:26.647507 1175 rpcz_store.cc:276] 0112 02:52:25.506101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:25.508475 (+ 2374us) service_pool.cc:225] Handling call
0112 02:52:26.645876 (+1137401us) ts_tablet_manager.cc:1145] Deleting tablet cb724feccc7f4fbc846fb6fa61be3385
0112 02:52:26.645887 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:26.647302 (+ 1415us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1137177}
W20250112 02:52:26.647909 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet cb724feccc7f4fbc846fb6fa61be3385 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb724feccc7f4fbc846fb6fa61be3385 already in progress: creating tablet
I20250112 02:52:26.651618 1175 tablet_replica.cc:331] T 49650a419dfd49d5a1547770d3f2fe54 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:26.651830 1175 raft_consensus.cc:2238] T 49650a419dfd49d5a1547770d3f2fe54 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:26.652002 1175 raft_consensus.cc:2267] T 49650a419dfd49d5a1547770d3f2fe54 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.652631 1175 ts_tablet_manager.cc:1905] T 49650a419dfd49d5a1547770d3f2fe54 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.652500 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_6f3879f2469d4655a6d4436362d0c3bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.653007 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f3879f2469d4655a6d4436362d0c3bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.654186 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet fb0aba0139cf479b80b32f3dfc936535
I20250112 02:52:26.654444 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 61593969d2234ce3a6089027b067d922 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.654743 518 catalog_manager.cc:6596] ReplaceTablet: tablet fb0aba0139cf479b80b32f3dfc936535 deleted and replaced by tablet e2f1f1a76e5f4308bf56dd647f02c6d8
I20250112 02:52:26.654731 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a18132549374218893f35a4e4226b72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.655047 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_a559cd55c5dc4019975f14dd54fe8589"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.655529 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a559cd55c5dc4019975f14dd54fe8589 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.656073 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu as pid 1229
I20250112 02:52:26.657006 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_105e5d1a8ef846ffbcb1bdbe01bbae55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.657625 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_105e5d1a8ef846ffbcb1bdbe01bbae55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:26.659152 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe614eaeb793473d9b2da206b7345562 has no consensus state
I20250112 02:52:26.660583 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_e1d4a3104fc0481287acede4261941c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.661000 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1d4a3104fc0481287acede4261941c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.662253 757 tablet_service.cc:1514] Processing DeleteTablet for tablet f7cd0927c6a4401cbe69f55d4db9963a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.662380 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b885247d2cb449e0a29f6f5d81518589" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:26.662518 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 54033a747d5a49b0a1c16f138158f596 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.663951 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 8a12b225131b464494a9f02133d37aea
I20250112 02:52:26.664559 524 catalog_manager.cc:6596] ReplaceTablet: tablet 8a12b225131b464494a9f02133d37aea deleted and replaced by tablet d45fce86e02f46d4a57af718fa2a62c8
I20250112 02:52:26.664914 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_ad5f01f6a5274cc483c8f051059d36c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.665390 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad5f01f6a5274cc483c8f051059d36c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.672545 1181 tablet_bootstrap.cc:492] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:26.672686 1181 ts_tablet_manager.cc:1397] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.005s
I20250112 02:52:26.659438 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_f7f876c63e68490c805c0bfa3f0c25ea" } new_table_name: "default.Table_fb003efcf38c4d18897952ba1cfe5ca8" modify_external_catalogs: true
I20250112 02:52:26.673656 1181 raft_consensus.cc:357] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.673884 1181 raft_consensus.cc:383] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.673982 1181 raft_consensus.cc:738] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.674232 1181 consensus_queue.cc:260] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.674731 1181 ts_tablet_manager.cc:1428] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:26.675108 1181 tablet_bootstrap.cc:492] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:26.675132 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet c2595782d66e4e4b890c94221b196daa (table default.Table_96a35d4e94284a49ba1a88d7d0acd3c1 [id=8559c454b17b4a00a60689a6a9b19f7d]) successfully deleted
I20250112 02:52:26.675714 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b885247d2cb449e0a29f6f5d81518589" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:26.675895 891 tablet_service.cc:1514] Processing DeleteTablet for tablet fb0aba0139cf479b80b32f3dfc936535 with delete_type TABLET_DATA_DELETED (replaced by tablet e2f1f1a76e5f4308bf56dd647f02c6d8) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.676090 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a12b225131b464494a9f02133d37aea with delete_type TABLET_DATA_DELETED (replaced by tablet d45fce86e02f46d4a57af718fa2a62c8) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.676254 760 tablet_service.cc:1514] Processing DeleteTablet for tablet fb0aba0139cf479b80b32f3dfc936535 with delete_type TABLET_DATA_DELETED (replaced by tablet e2f1f1a76e5f4308bf56dd647f02c6d8) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.676378 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69587c5a09d40f0baba5c2f6e8a7b41" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:26.676440 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a12b225131b464494a9f02133d37aea with delete_type TABLET_DATA_DELETED (replaced by tablet d45fce86e02f46d4a57af718fa2a62c8) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:26.676775 854 leader_election.cc:343] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.676389 1173 tablet_bootstrap.cc:492] T 96e201d798c94e448a78e1c8adf41724 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
W20250112 02:52:26.677018 854 leader_election.cc:343] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.677249 1269 raft_consensus.cc:491] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:26.677362 1269 raft_consensus.cc:513] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.677938 1269 leader_election.cc:290] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:26.677958 1173 ts_tablet_manager.cc:1397] T 96e201d798c94e448a78e1c8adf41724 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.096s user 0.004s sys 0.000s
I20250112 02:52:26.678814 1181 tablet_bootstrap.cc:654] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.684367 1440 sys_catalog.cc:263] Verifying existing consensus state
I20250112 02:52:26.684621 1173 raft_consensus.cc:357] T 96e201d798c94e448a78e1c8adf41724 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.684828 1173 raft_consensus.cc:383] T 96e201d798c94e448a78e1c8adf41724 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.688825 1440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304: Bootstrap starting.
I20250112 02:52:26.690415 1175 ts_tablet_manager.cc:1918] T 49650a419dfd49d5a1547770d3f2fe54 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:26.690546 1175 log.cc:1198] T 49650a419dfd49d5a1547770d3f2fe54 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/49650a419dfd49d5a1547770d3f2fe54
I20250112 02:52:26.690833 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9186ac46ed441ee9308490e90da0e23" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:26.691648 855 leader_election.cc:343] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.691782 855 leader_election.cc:304] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:26.692035 1269 raft_consensus.cc:2743] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:26.692817 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9186ac46ed441ee9308490e90da0e23" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:26.693118 1172 ts_tablet_manager.cc:1918] T 09886cd08c714f24a96e4df02bc0e6c8 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:26.694159 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69587c5a09d40f0baba5c2f6e8a7b41" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:26.694274 854 leader_election.cc:343] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:26.694396 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ee98f055024429eafaa29bb961cab87 consensus state has no leader
I20250112 02:52:26.694645 1172 log.cc:1198] T 09886cd08c714f24a96e4df02bc0e6c8 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/09886cd08c714f24a96e4df02bc0e6c8
W20250112 02:52:26.694698 855 leader_election.cc:343] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.694800 855 leader_election.cc:304] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:26.694944 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 23e7a2e88c374e949ced92c557225cbf consensus state has no leader
W20250112 02:52:26.695183 855 leader_election.cc:343] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.695276 855 leader_election.cc:304] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:26.695459 1172 ts_tablet_manager.cc:1939] T 09886cd08c714f24a96e4df02bc0e6c8 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:26.696050 1190 raft_consensus.cc:2743] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:26.696216 1190 raft_consensus.cc:2743] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:26.697304 1173 raft_consensus.cc:738] T 96e201d798c94e448a78e1c8adf41724 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
W20250112 02:52:26.699820 855 consensus_peers.cc:487] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2595782d66e4e4b890c94221b196daa. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:26.701175 1171 tablet_replica.cc:331] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
W20250112 02:52:26.701269 513 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 49650a419dfd49d5a1547770d3f2fe54 because the tablet was not found. No further retry: Not found: Tablet not found: 49650a419dfd49d5a1547770d3f2fe54
I20250112 02:52:26.701427 1171 raft_consensus.cc:2238] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:26.701589 1171 raft_consensus.cc:2267] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.702087 1171 ts_tablet_manager.cc:1905] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.705011 1173 consensus_queue.cc:260] T 96e201d798c94e448a78e1c8adf41724 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.705583 1173 ts_tablet_manager.cc:1428] T 96e201d798c94e448a78e1c8adf41724 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250112 02:52:26.705924 1173 tablet_bootstrap.cc:492] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:26.708022 519 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 96e201d798c94e448a78e1c8adf41724 (table default.Table_4c59f5d61aab4b6184f0d9a4e21f965f [id=95cdab3e555643fc990b77b2ee8f058a]). Expected version 1 got 0
I20250112 02:52:26.708781 1172 tablet_replica.cc:331] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:26.708963 1172 raft_consensus.cc:2238] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:26.709153 1172 raft_consensus.cc:2267] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.710124 1172 ts_tablet_manager.cc:1905] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.710583 519 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 96e201d798c94e448a78e1c8adf41724 (table default.Table_4c59f5d61aab4b6184f0d9a4e21f965f [id=95cdab3e555643fc990b77b2ee8f058a]). Expected version 1 got 0
I20250112 02:52:26.710664 1173 tablet_bootstrap.cc:654] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.717067 1172 ts_tablet_manager.cc:1918] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:26.717156 1049 raft_consensus.cc:3054] T 96e201d798c94e448a78e1c8adf41724 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:26.717185 1172 log.cc:1198] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/f05d3326545849b9af4519bc66c21069
I20250112 02:52:26.717602 1172 ts_tablet_manager.cc:1939] T f05d3326545849b9af4519bc66c21069 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:26.728755 1177 tablet_bootstrap.cc:492] T fb4c004397ee480d897e8b125d248829 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:26.729070 1177 ts_tablet_manager.cc:1397] T fb4c004397ee480d897e8b125d248829 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.131s user 0.002s sys 0.007s
I20250112 02:52:26.732424 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet fb0aba0139cf479b80b32f3dfc936535 with delete_type TABLET_DATA_DELETED (replaced by tablet e2f1f1a76e5f4308bf56dd647f02c6d8) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.732627 1173 tablet_bootstrap.cc:492] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:26.732848 1173 ts_tablet_manager.cc:1397] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20250112 02:52:26.733103 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a12b225131b464494a9f02133d37aea with delete_type TABLET_DATA_DELETED (replaced by tablet d45fce86e02f46d4a57af718fa2a62c8) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.733819 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 09886cd08c714f24a96e4df02bc0e6c8 (table default.Table_f82a967adb9040f69df9732d0d228d8d [id=fce07a88937f4f5d8b14483e056bcb4d]) successfully deleted
I20250112 02:52:26.733795 1173 raft_consensus.cc:357] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.734052 1173 raft_consensus.cc:383] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.734520 1173 raft_consensus.cc:738] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.734802 1173 consensus_queue.cc:260] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.736538 1173 ts_tablet_manager.cc:1428] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 02:52:26.736789 1173 tablet_bootstrap.cc:492] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:26.730309 1177 raft_consensus.cc:357] T fb4c004397ee480d897e8b125d248829 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.740365 1177 raft_consensus.cc:383] T fb4c004397ee480d897e8b125d248829 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.740475 1177 raft_consensus.cc:738] T fb4c004397ee480d897e8b125d248829 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.744529 519 master_service.cc:946] ReplaceTablet: received request to replace tablet b7bd96fb381f4e73a0081be3fc799c36 from {username='slave'} at 127.0.0.1:40934
W20250112 02:52:26.745361 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 64707ee432fb4e96a3bbf7252dfa1813 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64707ee432fb4e96a3bbf7252dfa1813 already in progress: creating tablet
I20250112 02:52:26.745731 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet f05d3326545849b9af4519bc66c21069 (table default.Table_15a37175ae434dfc9638765263dfb60f [id=7acec972e5f54ed7b0ae6f3c8d317b68]) successfully deleted
W20250112 02:52:26.746011 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 61593969d2234ce3a6089027b067d922 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61593969d2234ce3a6089027b067d922 already in progress: creating tablet
W20250112 02:52:26.746354 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 9d5b3df88a0c45dba879b3d506710614 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d5b3df88a0c45dba879b3d506710614 already in progress: creating tablet
W20250112 02:52:26.746631 512 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f7cd0927c6a4401cbe69f55d4db9963a because the tablet was not found. No further retry: Not found: Tablet not found: f7cd0927c6a4401cbe69f55d4db9963a
W20250112 02:52:26.746868 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 7a18132549374218893f35a4e4226b72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a18132549374218893f35a4e4226b72 already in progress: creating tablet
W20250112 02:52:26.747097 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 54033a747d5a49b0a1c16f138158f596 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54033a747d5a49b0a1c16f138158f596 already in progress: creating tablet
I20250112 02:52:26.747591 1269 consensus_queue.cc:1035] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
I20250112 02:52:26.747822 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 235e9dd0964c4ea6af7d05ef58acae0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:23 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:26.748502 1190 raft_consensus.cc:491] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:26.748682 1190 raft_consensus.cc:513] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.749346 1190 leader_election.cc:290] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:26.747467 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet fb0aba0139cf479b80b32f3dfc936535 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb0aba0139cf479b80b32f3dfc936535 already in progress: creating tablet
I20250112 02:52:26.749979 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4c004397ee480d897e8b125d248829" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:26.750208 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 8a12b225131b464494a9f02133d37aea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a12b225131b464494a9f02133d37aea already in progress: creating tablet
I20250112 02:52:26.750799 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4c004397ee480d897e8b125d248829" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:26.751309 854 leader_election.cc:343] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.753933 1175 ts_tablet_manager.cc:1939] T 49650a419dfd49d5a1547770d3f2fe54 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:26.755275 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 194) took 1227 ms. Trace:
I20250112 02:52:26.755724 1172 tablet_replica.cc:331] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:26.756107 1172 raft_consensus.cc:2238] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:26.756536 1181 tablet_bootstrap.cc:492] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:26.756686 1181 ts_tablet_manager.cc:1397] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.082s user 0.002s sys 0.012s
I20250112 02:52:26.756873 1172 raft_consensus.cc:2267] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.756448 1175 rpcz_store.cc:276] 0112 02:52:25.527739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:25.527791 (+ 52us) service_pool.cc:225] Handling call
0112 02:52:26.651561 (+1123770us) ts_tablet_manager.cc:1145] Deleting tablet 49650a419dfd49d5a1547770d3f2fe54
0112 02:52:26.651570 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:26.660676 (+ 9106us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:26.690312 (+ 29636us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:26.755219 (+ 64907us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1123539}
I20250112 02:52:26.757429 1181 raft_consensus.cc:357] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.757583 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 49650a419dfd49d5a1547770d3f2fe54 (table default.Table_d561e1740cc545dc8aa4babb76b45c37 [id=de27aeff6b3a49b3956498ea5015ff33]) successfully deleted
I20250112 02:52:26.757619 1181 raft_consensus.cc:383] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.757675 1181 raft_consensus.cc:738] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.757911 1181 consensus_queue.cc:260] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.758446 1181 ts_tablet_manager.cc:1428] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:26.758760 1181 tablet_bootstrap.cc:492] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
W20250112 02:52:26.762530 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c03b5e612d64314b210c11686fdfa2a consensus state has no leader
I20250112 02:52:26.764400 1172 ts_tablet_manager.cc:1905] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.766757 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39987
I20250112 02:52:26.767774 757 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.768009 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 71b5f80ed198491780cbe17e6cd3fa15 with delete_type TABLET_DATA_DELETED (replaced by tablet c7c530aa0aa64e47ae27d61f341d2bd9) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.763651 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 200) took 1151 ms. Trace:
I20250112 02:52:26.768146 1175 rpcz_store.cc:276] 0112 02:52:25.611779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:25.611959 (+ 180us) service_pool.cc:225] Handling call
0112 02:52:26.763507 (+1151548us) ts_tablet_manager.cc:1145] Deleting tablet ea33e5c1a1fb46599c5836a36b130051
0112 02:52:26.763516 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:26.763604 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1151320}
W20250112 02:52:26.768548 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet ea33e5c1a1fb46599c5836a36b130051 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea33e5c1a1fb46599c5836a36b130051 already in progress: creating tablet
I20250112 02:52:26.770324 1173 tablet_bootstrap.cc:654] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.771363 1175 tablet_replica.cc:331] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:26.768854 1177 consensus_queue.cc:260] T fb4c004397ee480d897e8b125d248829 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.771869 780 raft_consensus.cc:2463] T fb4c004397ee480d897e8b125d248829 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:26.772600 855 leader_election.cc:304] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:26.773161 1190 raft_consensus.cc:2798] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:26.773269 1190 raft_consensus.cc:491] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:26.773344 1190 raft_consensus.cc:3054] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:26.773223 517 catalog_manager.cc:5526] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:26.778091 1190 raft_consensus.cc:513] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.778986 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4c004397ee480d897e8b125d248829" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:26.779196 780 raft_consensus.cc:3054] T fb4c004397ee480d897e8b125d248829 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:26.781203 893 tablet_service.cc:1467] Processing CreateTablet for tablet e933b58dab0b45e8bf1c49eebb1c53e3 (DEFAULT_TABLE table=default.Table_46a03fa2d75e4541a98a1b3d91f85395 [id=e184c10af82240f5b1e29fc226269bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.781606 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e933b58dab0b45e8bf1c49eebb1c53e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.781726 780 raft_consensus.cc:2463] T fb4c004397ee480d897e8b125d248829 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:26.781852 891 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb76746b3a846c3aba531c04bf63966 (DEFAULT_TABLE table=default.Table_75db5f75b2874392b44dd580e0e52de9 [id=013bd2f66fe64ed593d8c0261ffa1c18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.782166 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cb76746b3a846c3aba531c04bf63966. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.782125 855 leader_election.cc:304] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:26.782397 1269 raft_consensus.cc:2798] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:26.783773 897 tablet_service.cc:1467] Processing CreateTablet for tablet fd7394b1878447a189ce45957f10e4df (DEFAULT_TABLE table=default.Table_3d820aeff9534cc98f5722887aed5300 [id=08a760dcd20642fb869437d4ccc3f333]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.784144 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd7394b1878447a189ce45957f10e4df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.784523 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4c004397ee480d897e8b125d248829" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:26.784971 854 leader_election.cc:343] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.785285 1190 leader_election.cc:290] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:26.785980 886 tablet_service.cc:1467] Processing CreateTablet for tablet b8b64e4e64cc462891f1ee7a594d5026 (DEFAULT_TABLE table=default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.786362 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b64e4e64cc462891f1ee7a594d5026. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.786527 1181 tablet_bootstrap.cc:654] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.788079 888 tablet_service.cc:1467] Processing CreateTablet for tablet 4d25815a5df94e099e7481c48c3da793 (DEFAULT_TABLE table=default.Table_dd5ce8476cb84204b0351fd07cab2b00 [id=b6d8e3bccc9c4cc694593463768a7e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.788430 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d25815a5df94e099e7481c48c3da793. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.789072 756 tablet_service.cc:1467] Processing CreateTablet for tablet e933b58dab0b45e8bf1c49eebb1c53e3 (DEFAULT_TABLE table=default.Table_46a03fa2d75e4541a98a1b3d91f85395 [id=e184c10af82240f5b1e29fc226269bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.789274 758 tablet_service.cc:1467] Processing CreateTablet for tablet 4d25815a5df94e099e7481c48c3da793 (DEFAULT_TABLE table=default.Table_dd5ce8476cb84204b0351fd07cab2b00 [id=b6d8e3bccc9c4cc694593463768a7e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.789484 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e933b58dab0b45e8bf1c49eebb1c53e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.789666 754 tablet_service.cc:1467] Processing CreateTablet for tablet 87759e8b99b04e628badc8f60cb38bb3 (DEFAULT_TABLE table=default.Table_abf2c7b9b8da40c6b9a8381b4e41bda6 [id=4a81fc17cbb94a23a53b5430b545a756]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.789754 750 tablet_service.cc:1467] Processing CreateTablet for tablet b8b64e4e64cc462891f1ee7a594d5026 (DEFAULT_TABLE table=default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.790020 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87759e8b99b04e628badc8f60cb38bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.790230 751 tablet_service.cc:1467] Processing CreateTablet for tablet fe614eaeb793473d9b2da206b7345562 (DEFAULT_TABLE table=default.Table_b2b3c738a24f451eac39fb1a6399f37a [id=7c6c2713869e476cb23d157a285a5afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.790555 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe614eaeb793473d9b2da206b7345562. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.791685 892 tablet_service.cc:1467] Processing CreateTablet for tablet 87759e8b99b04e628badc8f60cb38bb3 (DEFAULT_TABLE table=default.Table_abf2c7b9b8da40c6b9a8381b4e41bda6 [id=4a81fc17cbb94a23a53b5430b545a756]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.791729 753 tablet_service.cc:1467] Processing CreateTablet for tablet 5d0c3191973a477e949fbf9dfcca963f (DEFAULT_TABLE table=default.Table_2740664d25a8451b87667c0eaf8b4d1b [id=2f43ebd749304c0287769224ac6b12c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.792016 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87759e8b99b04e628badc8f60cb38bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.792081 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d0c3191973a477e949fbf9dfcca963f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.792662 749 tablet_service.cc:1467] Processing CreateTablet for tablet fd7394b1878447a189ce45957f10e4df (DEFAULT_TABLE table=default.Table_3d820aeff9534cc98f5722887aed5300 [id=08a760dcd20642fb869437d4ccc3f333]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.792994 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd7394b1878447a189ce45957f10e4df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.795261 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b64e4e64cc462891f1ee7a594d5026. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.795228 898 tablet_service.cc:1467] Processing CreateTablet for tablet 5d0c3191973a477e949fbf9dfcca963f (DEFAULT_TABLE table=default.Table_2740664d25a8451b87667c0eaf8b4d1b [id=2f43ebd749304c0287769224ac6b12c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.795706 760 tablet_service.cc:1467] Processing CreateTablet for tablet d4abf59d96414e5faa7cf75ba133832c (DEFAULT_TABLE table=default.Table_525750dad2b7461586d070dbaa74d0b1 [id=7d048aa907a74605964b68726ac5d759]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.795874 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d0c3191973a477e949fbf9dfcca963f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.796197 757 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb76746b3a846c3aba531c04bf63966 (DEFAULT_TABLE table=default.Table_75db5f75b2874392b44dd580e0e52de9 [id=013bd2f66fe64ed593d8c0261ffa1c18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.781608 896 tablet_service.cc:1467] Processing CreateTablet for tablet d4abf59d96414e5faa7cf75ba133832c (DEFAULT_TABLE table=default.Table_525750dad2b7461586d070dbaa74d0b1 [id=7d048aa907a74605964b68726ac5d759]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.797763 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4abf59d96414e5faa7cf75ba133832c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.799589 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d25815a5df94e099e7481c48c3da793. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.800144 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cb76746b3a846c3aba531c04bf63966. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.804692 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4abf59d96414e5faa7cf75ba133832c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.809036 521 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_44769a7c8b7e463dbe5d907dacc0410f" } modify_external_catalogs: true
I20250112 02:52:26.809304 521 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_44769a7c8b7e463dbe5d907dacc0410f" } modify_external_catalogs: true
I20250112 02:52:26.811691 526 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_feacb3254bbe4765a4e0a4908fbc694c" } modify_external_catalogs: true
I20250112 02:52:26.811892 526 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_feacb3254bbe4765a4e0a4908fbc694c" } modify_external_catalogs: true
I20250112 02:52:26.812597 1440 log.cc:826] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304: Log is configured to *not* fsync() on all Append() calls
I20250112 02:52:26.813293 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_178713b464bb4f16970851646a710574"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.813771 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_178713b464bb4f16970851646a710574 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.814677 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_247a821819994de6adac6fabd06c3a1f" } modify_external_catalogs: true
I20250112 02:52:26.814790 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_247a821819994de6adac6fabd06c3a1f" } modify_external_catalogs: true
I20250112 02:52:26.815749 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_32538719e60a4843b560a8a6f3ac17b1" } new_table_name: "default.Table_69f363c4da234792886b544823b3214e" modify_external_catalogs: true
I20250112 02:52:26.816519 1175 raft_consensus.cc:2238] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:26.816712 1175 raft_consensus.cc:2267] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.820597 1177 ts_tablet_manager.cc:1428] T fb4c004397ee480d897e8b125d248829 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.091s user 0.002s sys 0.000s
I20250112 02:52:26.820863 892 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a0a13696fc41ccb28218f1275a4b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.821101 897 tablet_service.cc:1514] Processing DeleteTablet for tablet b885247d2cb449e0a29f6f5d81518589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.823055 825 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39987
I20250112 02:52:26.823367 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_6ff2131cc2144af7b6c223f14e9eb25a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.823815 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ff2131cc2144af7b6c223f14e9eb25a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.827803 1269 raft_consensus.cc:695] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:26.828119 1269 consensus_queue.cc:237] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.829639 1177 tablet_bootstrap.cc:492] T 9427951452104a7db765437493b21e90 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:26.831270 1177 tablet_bootstrap.cc:654] T 9427951452104a7db765437493b21e90 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.831910 1175 ts_tablet_manager.cc:1905] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.833300 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_a559cd55c5dc4019975f14dd54fe8589" } modify_external_catalogs: true
I20250112 02:52:26.833457 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_a559cd55c5dc4019975f14dd54fe8589" } modify_external_catalogs: true
I20250112 02:52:26.835047 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_7e09abe4b91b4b2db2bf7d3085e19502" } modify_external_catalogs: true
I20250112 02:52:26.835314 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_7e09abe4b91b4b2db2bf7d3085e19502" } modify_external_catalogs: true
W20250112 02:52:26.836380 512 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:42341, user_credentials={real_user=slave}} blocked reactor thread for 77738us
I20250112 02:52:26.845436 1173 tablet_bootstrap.cc:492] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:26.845579 1173 ts_tablet_manager.cc:1397] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.109s user 0.001s sys 0.002s
I20250112 02:52:26.846385 1173 raft_consensus.cc:357] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.846601 1173 raft_consensus.cc:383] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.846688 1173 raft_consensus.cc:738] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.846726 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet c7c530aa0aa64e47ae27d61f341d2bd9
I20250112 02:52:26.846906 1173 consensus_queue.cc:260] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.848449 526 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 4b599e983ec04f53a440d1ab828d12ab
I20250112 02:52:26.850066 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet b9199b0d2b924a79938a979df4196bb4
I20250112 02:52:26.853250 1173 ts_tablet_manager.cc:1428] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250112 02:52:26.853509 1173 tablet_bootstrap.cc:492] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:26.855190 1173 tablet_bootstrap.cc:654] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.856990 888 tablet_service.cc:1514] Processing DeleteTablet for tablet c7c530aa0aa64e47ae27d61f341d2bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.860152 1172 ts_tablet_manager.cc:1918] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:26.860996 888 tablet_service.cc:1467] Processing CreateTablet for tablet fe614eaeb793473d9b2da206b7345562 (DEFAULT_TABLE table=default.Table_b2b3c738a24f451eac39fb1a6399f37a [id=7c6c2713869e476cb23d157a285a5afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.861353 756 tablet_service.cc:1514] Processing DeleteTablet for tablet d3e41e307df64109a41b0c1c44d3bda6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.861949 897 tablet_service.cc:1514] Processing DeleteTablet for tablet b9199b0d2b924a79938a979df4196bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.862202 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe614eaeb793473d9b2da206b7345562. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.862313 891 tablet_service.cc:1467] Processing CreateTablet for tablet b9537fcd96eb41cc8f680075ec71137b (DEFAULT_TABLE table=default.Table_f22fde4d1df34a7b955de7c6eee20c3a [id=485dd281891f45e8b407b306572d6e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.862674 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9537fcd96eb41cc8f680075ec71137b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.863349 758 tablet_service.cc:1514] Processing DeleteTablet for tablet b9199b0d2b924a79938a979df4196bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.863701 751 tablet_service.cc:1514] Processing DeleteTablet for tablet c7c530aa0aa64e47ae27d61f341d2bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.864005 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b599e983ec04f53a440d1ab828d12ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.864374 1440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 10 replicates
I20250112 02:52:26.865048 1440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304: Bootstrap complete.
I20250112 02:52:26.865491 758 tablet_service.cc:1467] Processing CreateTablet for tablet 416fb6820cd54e118950c68d381f51b2 (DEFAULT_TABLE table=default.Table_f4d612cd6a524bc493147493c5a4a284 [id=4ac16bc62bbe4ed7a977d22c92ad580d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.865828 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b599e983ec04f53a440d1ab828d12ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:26.866233 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 416fb6820cd54e118950c68d381f51b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.867365 751 tablet_service.cc:1467] Processing CreateTablet for tablet b3c5a861a90840d0a33b780ec75bffd2 (DEFAULT_TABLE table=default.Table_c26869f6cbd540fe980dcf207b5c729d [id=cdd6a2097792484d84bca856f30d7ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.871369 750 tablet_service.cc:1467] Processing CreateTablet for tablet a12c5d58e6c246c983e8a533906ce77d (DEFAULT_TABLE table=default.Table_f7bab317410d408aa363857a99be6e3c [id=d0be0fa53a79498e9160bfaa7d88837b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.871769 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3c5a861a90840d0a33b780ec75bffd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.872879 896 tablet_service.cc:1467] Processing CreateTablet for tablet a12c5d58e6c246c983e8a533906ce77d (DEFAULT_TABLE table=default.Table_f7bab317410d408aa363857a99be6e3c [id=d0be0fa53a79498e9160bfaa7d88837b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.876168 892 tablet_service.cc:1467] Processing CreateTablet for tablet 416fb6820cd54e118950c68d381f51b2 (DEFAULT_TABLE table=default.Table_f4d612cd6a524bc493147493c5a4a284 [id=4ac16bc62bbe4ed7a977d22c92ad580d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.876616 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a12c5d58e6c246c983e8a533906ce77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.877023 1449 consensus_queue.cc:1035] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:26.877663 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a12c5d58e6c246c983e8a533906ce77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.878327 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 416fb6820cd54e118950c68d381f51b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.878674 1175 ts_tablet_manager.cc:1918] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:26.878778 1175 log.cc:1198] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/df6ccd04880442d0990c6cb0d27956bd
I20250112 02:52:26.876158 754 tablet_service.cc:1467] Processing CreateTablet for tablet b9537fcd96eb41cc8f680075ec71137b (DEFAULT_TABLE table=default.Table_f22fde4d1df34a7b955de7c6eee20c3a [id=485dd281891f45e8b407b306572d6e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.884164 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_4657847b84384e628194a8ff43e183a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.884645 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4657847b84384e628194a8ff43e183a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.886188 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_8dd5b6fa78f64fe78de385801d15fafa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.886642 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8dd5b6fa78f64fe78de385801d15fafa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.887928 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_952b91d2b0b8435886e555f12629dc80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.888409 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_952b91d2b0b8435886e555f12629dc80 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.892854 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_a60b0bb4ae784e09bb62235bfb1d7bcd" } new_table_name: "default.Table_f7d4187a2ec34710a2e1c52c00be1fb8" modify_external_catalogs: true
I20250112 02:52:26.894131 1294 raft_consensus.cc:491] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:26.894277 1294 raft_consensus.cc:513] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:26.894834 1294 leader_election.cc:290] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:26.894915 757 tablet_service.cc:1514] Processing DeleteTablet for tablet fb4c004397ee480d897e8b125d248829 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:26.896908 1449 raft_consensus.cc:491] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 02:52:26.896935 713 leader_election.cc:343] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 71b5f80ed198491780cbe17e6cd3fa15
I20250112 02:52:26.899281 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_548027a9e05c4376b024df028f14e86d" } new_table_name: "default.Table_a98f3e8602ec4ba4a771b14df9c07131" modify_external_catalogs: true
W20250112 02:52:26.904359 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4916ae3917084cfe8c93240a8857be0e consensus state has no leader
I20250112 02:52:26.905310 1440 raft_consensus.cc:357] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 3 FOLLOWER]: Replica starting. Triggering 10 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:26.909495 523 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 96e201d798c94e448a78e1c8adf41724 (table default.Table_4c59f5d61aab4b6184f0d9a4e21f965f [id=95cdab3e555643fc990b77b2ee8f058a]). Expected version 1 got 0
I20250112 02:52:26.909915 1175 ts_tablet_manager.cc:1939] T df6ccd04880442d0990c6cb0d27956bd P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:26.909950 897 tablet_service.cc:1467] Processing CreateTablet for tablet b3c5a861a90840d0a33b780ec75bffd2 (DEFAULT_TABLE table=default.Table_c26869f6cbd540fe980dcf207b5c729d [id=cdd6a2097792484d84bca856f30d7ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:26.910399 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3c5a861a90840d0a33b780ec75bffd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.910490 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 199) took 1298 ms. Trace:
I20250112 02:52:26.910580 1175 rpcz_store.cc:276] 0112 02:52:25.611678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:25.612212 (+ 534us) service_pool.cc:225] Handling call
0112 02:52:26.771287 (+1159075us) ts_tablet_manager.cc:1145] Deleting tablet df6ccd04880442d0990c6cb0d27956bd
0112 02:52:26.771297 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:26.834434 (+ 63137us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:26.878564 (+ 44130us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:26.910457 (+ 31893us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1158910}
I20250112 02:52:26.910951 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet df6ccd04880442d0990c6cb0d27956bd (table default.Table_c667abf579554ea0af764de6515a18c6 [id=a8c0e9da5acc45c3ab7f67427f59e2b6]) successfully deleted
I20250112 02:52:26.911404 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 198) took 1299 ms. Trace:
I20250112 02:52:26.911603 1175 rpcz_store.cc:276] 0112 02:52:25.611549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:25.627863 (+ 16314us) service_pool.cc:225] Handling call
0112 02:52:26.911265 (+1283402us) ts_tablet_manager.cc:1145] Deleting tablet 49bb6d20cfa0453e90e048825114525e
0112 02:52:26.911273 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:26.911370 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1283201}
I20250112 02:52:26.911953 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_0f67e34fb9604f57bbf8656d531ce52c" } new_table_name: "default.Table_4603068c01544e2e965dca5177e36b40" modify_external_catalogs: true
W20250112 02:52:26.912086 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 49bb6d20cfa0453e90e048825114525e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49bb6d20cfa0453e90e048825114525e already in progress: creating tablet
I20250112 02:52:26.912380 1175 tablet_replica.cc:331] T 4bc067e895cb4c69af4dedc03533a276 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:26.912568 1175 raft_consensus.cc:2238] T 4bc067e895cb4c69af4dedc03533a276 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:26.912731 1175 raft_consensus.cc:2267] T 4bc067e895cb4c69af4dedc03533a276 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.913218 1175 ts_tablet_manager.cc:1905] T 4bc067e895cb4c69af4dedc03533a276 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.915526 1449 raft_consensus.cc:513] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.916218 1449 leader_election.cc:290] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:26.926800 519 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 7c1c45b2e3d040beaa69fa1b6b306eb9
I20250112 02:52:26.926805 1175 ts_tablet_manager.cc:1918] T 4bc067e895cb4c69af4dedc03533a276 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:26.926973 1175 log.cc:1198] T 4bc067e895cb4c69af4dedc03533a276 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/4bc067e895cb4c69af4dedc03533a276
I20250112 02:52:26.927386 1175 ts_tablet_manager.cc:1939] T 4bc067e895cb4c69af4dedc03533a276 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:26.927397 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9537fcd96eb41cc8f680075ec71137b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:26.927752 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 201) took 1280 ms. Trace:
I20250112 02:52:26.927862 1175 rpcz_store.cc:276] 0112 02:52:25.647007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:25.647498 (+ 491us) service_pool.cc:225] Handling call
0112 02:52:26.912321 (+1264823us) ts_tablet_manager.cc:1145] Deleting tablet 4bc067e895cb4c69af4dedc03533a276
0112 02:52:26.912330 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:26.916213 (+ 3883us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:26.926533 (+ 10320us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:26.927697 (+ 1164us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1260610}
I20250112 02:52:26.928323 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 4bc067e895cb4c69af4dedc03533a276 (table default.Table_6e85e49c707f4021b9a16ca854bb2634 [id=03ca66b32af3425892725946ed90c493]) successfully deleted
I20250112 02:52:26.928479 1175 tablet_replica.cc:331] T 3567820f0d4349799b681fdfff0ebad8 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:26.928287 525 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 4e11086ad9b4468baf21beedcde2e952
I20250112 02:52:26.928673 1175 raft_consensus.cc:2238] T 3567820f0d4349799b681fdfff0ebad8 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:26.928861 1175 raft_consensus.cc:2267] T 3567820f0d4349799b681fdfff0ebad8 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.929594 1175 ts_tablet_manager.cc:1905] T 3567820f0d4349799b681fdfff0ebad8 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.934085 1171 ts_tablet_manager.cc:1918] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:26.934238 1171 log.cc:1198] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/328e307a4b35445f92cbfab24f2ada48
I20250112 02:52:26.934623 1171 ts_tablet_manager.cc:1939] T 328e307a4b35445f92cbfab24f2ada48 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:26.935526 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 328e307a4b35445f92cbfab24f2ada48 (table default.Table_fc5d684c7f2049e5b2976823cfb242ae [id=c84a7bdac0ed4c2ea7c0599d5f1ca9c1]) successfully deleted
W20250112 02:52:26.940634 513 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet aa67bf25dc894d7ea96d247a22c14e03 because the tablet was not found. No further retry: Not found: Tablet not found: aa67bf25dc894d7ea96d247a22c14e03
I20250112 02:52:26.941875 1269 raft_consensus.cc:491] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:26.942119 1269 raft_consensus.cc:513] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.946105 1171 tablet_replica.cc:331] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:26.946348 1171 raft_consensus.cc:2238] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:26.946609 1171 pending_rounds.cc:62] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:26.946734 1171 pending_rounds.cc:69] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319817620983808 op_type: NO_OP noop_request { }
I20250112 02:52:26.946913 1171 raft_consensus.cc:2883] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:26.947103 1171 raft_consensus.cc:2267] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.949107 1269 leader_election.cc:290] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:26.953047 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_af61980bca5e4995a77da652dcfb3f0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:26.954315 1171 ts_tablet_manager.cc:1905] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.964462 1440 raft_consensus.cc:738] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd25f03f25df49178a5c53c636ecf304, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.965633 1181 tablet_bootstrap.cc:492] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:26.965780 1181 ts_tablet_manager.cc:1397] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.207s user 0.019s sys 0.020s
I20250112 02:52:26.965644 1175 ts_tablet_manager.cc:1918] T 3567820f0d4349799b681fdfff0ebad8 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:26.966030 1175 log.cc:1198] T 3567820f0d4349799b681fdfff0ebad8 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/3567820f0d4349799b681fdfff0ebad8
I20250112 02:52:26.966593 1181 raft_consensus.cc:357] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.966779 1181 raft_consensus.cc:383] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.966871 1181 raft_consensus.cc:738] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.967128 1181 consensus_queue.cc:260] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.968178 1177 tablet_bootstrap.cc:492] T 9427951452104a7db765437493b21e90 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:26.968344 1177 ts_tablet_manager.cc:1397] T 9427951452104a7db765437493b21e90 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.139s user 0.004s sys 0.002s
I20250112 02:52:26.969019 1177 raft_consensus.cc:357] T 9427951452104a7db765437493b21e90 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.969281 1177 raft_consensus.cc:383] T 9427951452104a7db765437493b21e90 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.969365 1177 raft_consensus.cc:738] T 9427951452104a7db765437493b21e90 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.969616 1177 consensus_queue.cc:260] T 9427951452104a7db765437493b21e90 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.970625 1440 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:26.971064 1181 ts_tablet_manager.cc:1428] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:26.971736 1181 tablet_bootstrap.cc:492] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:26.972342 1175 ts_tablet_manager.cc:1939] T 3567820f0d4349799b681fdfff0ebad8 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:26.972884 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 209) took 1236 ms. Trace:
I20250112 02:52:26.973032 1175 rpcz_store.cc:276] 0112 02:52:25.736181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:25.736866 (+ 685us) service_pool.cc:225] Handling call
0112 02:52:26.928402 (+1191536us) ts_tablet_manager.cc:1145] Deleting tablet 3567820f0d4349799b681fdfff0ebad8
0112 02:52:26.928411 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:26.953897 (+ 25486us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:26.965254 (+ 11357us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:26.972821 (+ 7567us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1191356}
I20250112 02:52:26.973357 1181 tablet_bootstrap.cc:654] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.973863 1175 tablet_replica.cc:331] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:26.974169 1175 raft_consensus.cc:2238] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:26.974272 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_c734b496a0634288b6a93d8007262510" } new_table_name: "default.Table_365739770a11434dbf260ea912e7ea81" modify_external_catalogs: true
I20250112 02:52:26.974813 1175 raft_consensus.cc:2267] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:26.975538 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_dafd4ddc8ab34adfa999f9340216f765"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:26.976312 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_33b6530b130041f0a2a4a7240e92ff15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:26.976492 1175 ts_tablet_manager.cc:1905] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:26.977968 1462 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:26.978253 1462 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:52:26.978855 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_1a6ba73a53b14addbacc6a2ff0711256"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:26.982045 1181 tablet_bootstrap.cc:492] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:26.983315 1440 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 02:52:26.983440 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99715f37fbef47c6860fa1d95e350f20" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:26.983605 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dafd4ddc8ab34adfa999f9340216f765 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.983956 1181 ts_tablet_manager.cc:1397] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
W20250112 02:52:26.984568 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33b6530b130041f0a2a4a7240e92ff15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:26.984556 854 leader_election.cc:343] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.984697 1181 raft_consensus.cc:357] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.984887 1181 raft_consensus.cc:383] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.984962 1181 raft_consensus.cc:738] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.984978 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_7926c082d6c04315811ea3b4e1579e8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:26.985427 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7926c082d6c04315811ea3b4e1579e8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.986495 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6dc79f4d1194f81963a8094ed6b0d2e" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:26.987236 1181 consensus_queue.cc:260] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.987357 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99715f37fbef47c6860fa1d95e350f20" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:26.987568 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af61980bca5e4995a77da652dcfb3f0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.987895 1181 ts_tablet_manager.cc:1428] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20250112 02:52:26.988514 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a6ba73a53b14addbacc6a2ff0711256 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:26.988548 1181 tablet_bootstrap.cc:492] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:26.990991 1181 tablet_bootstrap.cc:654] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:26.991510 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6dc79f4d1194f81963a8094ed6b0d2e" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:26.992376 854 leader_election.cc:343] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:26.996050 1181 tablet_bootstrap.cc:492] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:26.996196 1181 ts_tablet_manager.cc:1397] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20250112 02:52:26.996887 1181 raft_consensus.cc:357] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.997110 1181 raft_consensus.cc:383] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:26.997198 1181 raft_consensus.cc:738] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:26.997393 1181 consensus_queue.cc:260] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.997679 1302 raft_consensus.cc:491] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:26.997825 1181 ts_tablet_manager.cc:1428] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:26.997860 1302 raft_consensus.cc:513] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:26.998059 1181 tablet_bootstrap.cc:492] T e9fc9a4f1e8944abb76dad6c2ad53d35 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:26.998487 1302 leader_election.cc:290] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:26.999881 893 tablet_service.cc:1514] Processing DeleteTablet for tablet aba1744138fe42ae9d097db01cac083d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:27.001190 989 leader_election.cc:343] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: d3e41e307df64109a41b0c1c44d3bda6
I20250112 02:52:27.004395 1181 tablet_bootstrap.cc:654] T e9fc9a4f1e8944abb76dad6c2ad53d35 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:27.007776 1181 tablet_bootstrap.cc:492] T e9fc9a4f1e8944abb76dad6c2ad53d35 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:27.007892 1181 ts_tablet_manager.cc:1397] T e9fc9a4f1e8944abb76dad6c2ad53d35 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20250112 02:52:27.008322 521 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 96e201d798c94e448a78e1c8adf41724 (table default.Table_4c59f5d61aab4b6184f0d9a4e21f965f [id=95cdab3e555643fc990b77b2ee8f058a]). Expected version 1 got 0
I20250112 02:52:27.008473 1181 raft_consensus.cc:357] T e9fc9a4f1e8944abb76dad6c2ad53d35 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.008656 1181 raft_consensus.cc:383] T e9fc9a4f1e8944abb76dad6c2ad53d35 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.008704 1181 raft_consensus.cc:738] T e9fc9a4f1e8944abb76dad6c2ad53d35 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:27.008873 1181 consensus_queue.cc:260] T e9fc9a4f1e8944abb76dad6c2ad53d35 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.009282 1181 ts_tablet_manager.cc:1428] T e9fc9a4f1e8944abb76dad6c2ad53d35 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:27.009603 1181 tablet_bootstrap.cc:492] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:27.009747 893 tablet_service.cc:1514] Processing DeleteTablet for tablet b2d8ea2ef18a46508371b4789ab6f4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.010154 893 tablet_service.cc:1514] Processing DeleteTablet for tablet cb724feccc7f4fbc846fb6fa61be3385 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.013329 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e41e307df64109a41b0c1c44d3bda6" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:27.013607 779 raft_consensus.cc:2463] T d3e41e307df64109a41b0c1c44d3bda6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:27.016106 1181 tablet_bootstrap.cc:654] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:27.018072 855 leader_election.cc:343] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.018225 855 leader_election.cc:304] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:27.018580 855 leader_election.cc:343] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.018687 855 leader_election.cc:304] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:27.019733 1190 raft_consensus.cc:2743] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:27.019932 1190 raft_consensus.cc:2743] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:27.020200 1181 tablet_bootstrap.cc:492] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:27.020332 1181 ts_tablet_manager.cc:1397] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20250112 02:52:27.021996 1177 ts_tablet_manager.cc:1428] T 9427951452104a7db765437493b21e90 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.054s user 0.000s sys 0.002s
I20250112 02:52:27.022107 1181 raft_consensus.cc:357] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.022284 1181 raft_consensus.cc:383] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.022372 1181 raft_consensus.cc:738] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:27.022531 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 9427951452104a7db765437493b21e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:27.022543 1181 consensus_queue.cc:260] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.023023 1181 ts_tablet_manager.cc:1428] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 02:52:27.026181 991 leader_election.cc:304] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:27.026259 1181 tablet_bootstrap.cc:492] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:27.026683 1302 raft_consensus.cc:2798] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:27.026891 1302 raft_consensus.cc:491] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:27.026993 1302 raft_consensus.cc:3054] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:27.027942 1181 tablet_bootstrap.cc:654] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:27.029508 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab2e31a65294407bb62b13e92898a7d2 consensus state has no leader
W20250112 02:52:27.029856 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2730634a5cfd4c2fb5c4e30100c556d4 consensus state has no leader
I20250112 02:52:27.030262 1302 raft_consensus.cc:513] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.031275 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e41e307df64109a41b0c1c44d3bda6" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:27.031518 779 raft_consensus.cc:3054] T d3e41e307df64109a41b0c1c44d3bda6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:27.031905 1177 tablet_bootstrap.cc:492] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:27.032564 1181 tablet_bootstrap.cc:492] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:27.032689 1181 ts_tablet_manager.cc:1397] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20250112 02:52:27.033269 1181 raft_consensus.cc:357] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.033452 1181 raft_consensus.cc:383] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.033507 1181 raft_consensus.cc:738] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:27.033725 1181 consensus_queue.cc:260] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.034246 1181 ts_tablet_manager.cc:1428] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:27.036785 1181 tablet_bootstrap.cc:492] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:27.038372 1181 tablet_bootstrap.cc:654] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:27.038568 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f234256b343f447dadfc3d1936a824ee has no consensus state
I20250112 02:52:27.040493 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet d3e41e307df64109a41b0c1c44d3bda6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.043299 1181 tablet_bootstrap.cc:492] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:27.043434 1181 ts_tablet_manager.cc:1397] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20250112 02:52:27.044734 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb76746b3a846c3aba531c04bf63966 (DEFAULT_TABLE table=default.Table_75db5f75b2874392b44dd580e0e52de9 [id=013bd2f66fe64ed593d8c0261ffa1c18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.045131 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cb76746b3a846c3aba531c04bf63966. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.045182 1181 raft_consensus.cc:357] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.045341 1181 raft_consensus.cc:383] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.045416 1181 raft_consensus.cc:738] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:27.045619 1181 consensus_queue.cc:260] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.045917 1034 tablet_service.cc:1467] Processing CreateTablet for tablet e933b58dab0b45e8bf1c49eebb1c53e3 (DEFAULT_TABLE table=default.Table_46a03fa2d75e4541a98a1b3d91f85395 [id=e184c10af82240f5b1e29fc226269bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.046144 1181 ts_tablet_manager.cc:1428] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20250112 02:52:27.046245 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e933b58dab0b45e8bf1c49eebb1c53e3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:27.046242 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6117dc8b32a64535a2f4167a03bed0d2 consensus state has no leader
I20250112 02:52:27.047428 1023 tablet_service.cc:1467] Processing CreateTablet for tablet fd7394b1878447a189ce45957f10e4df (DEFAULT_TABLE table=default.Table_3d820aeff9534cc98f5722887aed5300 [id=08a760dcd20642fb869437d4ccc3f333]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.047780 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd7394b1878447a189ce45957f10e4df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.048906 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 4d25815a5df94e099e7481c48c3da793 (DEFAULT_TABLE table=default.Table_dd5ce8476cb84204b0351fd07cab2b00 [id=b6d8e3bccc9c4cc694593463768a7e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.049081 1181 tablet_bootstrap.cc:492] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:27.049217 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d25815a5df94e099e7481c48c3da793. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.049996 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 87759e8b99b04e628badc8f60cb38bb3 (DEFAULT_TABLE table=default.Table_abf2c7b9b8da40c6b9a8381b4e41bda6 [id=4a81fc17cbb94a23a53b5430b545a756]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.050163 520 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 96e201d798c94e448a78e1c8adf41724 (table default.Table_4c59f5d61aab4b6184f0d9a4e21f965f [id=95cdab3e555643fc990b77b2ee8f058a]). Expected version 1 got 0
I20250112 02:52:27.050345 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87759e8b99b04e628badc8f60cb38bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.050740 1181 tablet_bootstrap.cc:654] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:27.051229 1022 tablet_service.cc:1467] Processing CreateTablet for tablet b8b64e4e64cc462891f1ee7a594d5026 (DEFAULT_TABLE table=default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.051563 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b64e4e64cc462891f1ee7a594d5026. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.051663 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_3c1f0242d17443b5a758967cf82ca0ca" } modify_external_catalogs: true
I20250112 02:52:27.051832 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_3c1f0242d17443b5a758967cf82ca0ca" } modify_external_catalogs: true
I20250112 02:52:27.051970 1302 leader_election.cc:290] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:27.052783 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_105e5d1a8ef846ffbcb1bdbe01bbae55" } modify_external_catalogs: true
I20250112 02:52:27.052947 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_105e5d1a8ef846ffbcb1bdbe01bbae55" } modify_external_catalogs: true
I20250112 02:52:27.033598 1177 tablet_bootstrap.cc:654] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:27.054440 989 leader_election.cc:343] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: d3e41e307df64109a41b0c1c44d3bda6
I20250112 02:52:27.054574 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_679c07c6f25644c6a4136aa56dd5100a" } modify_external_catalogs: true
I20250112 02:52:27.054718 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_679c07c6f25644c6a4136aa56dd5100a" } modify_external_catalogs: true
I20250112 02:52:27.043815 1024 tablet_service.cc:1467] Processing CreateTablet for tablet d4abf59d96414e5faa7cf75ba133832c (DEFAULT_TABLE table=default.Table_525750dad2b7461586d070dbaa74d0b1 [id=7d048aa907a74605964b68726ac5d759]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.056771 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4abf59d96414e5faa7cf75ba133832c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:27.059087 513 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 8ee98f055024429eafaa29bb961cab87 (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8ee98f055024429eafaa29bb961cab87 consensus state has no leader
I20250112 02:52:27.059077 1171 ts_tablet_manager.cc:1918] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:27.059239 1171 log.cc:1198] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/3567820f0d4349799b681fdfff0ebad8
I20250112 02:52:27.059671 1171 ts_tablet_manager.cc:1939] T 3567820f0d4349799b681fdfff0ebad8 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:27.067148 1181 tablet_bootstrap.cc:492] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:27.067294 1181 ts_tablet_manager.cc:1397] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.002s
I20250112 02:52:27.063454 1177 tablet_bootstrap.cc:492] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:27.069207 1177 ts_tablet_manager.cc:1397] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.002s
I20250112 02:52:27.069336 1181 raft_consensus.cc:357] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.069607 1181 raft_consensus.cc:383] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.069694 1181 raft_consensus.cc:738] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:27.070046 1181 consensus_queue.cc:260] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.070384 1449 raft_consensus.cc:491] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:27.070603 1181 ts_tablet_manager.cc:1428] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 02:52:27.070545 1449 raft_consensus.cc:513] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.070885 1181 tablet_bootstrap.cc:492] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:27.071223 1449 leader_election.cc:290] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:27.072096 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a0a13696fc41ccb28218f1275a4b5d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:27.072659 855 leader_election.cc:343] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.073031 1181 tablet_bootstrap.cc:654] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:27.074024 1478 tablet.cc:1721] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:27.074501 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a0a13696fc41ccb28218f1275a4b5d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:27.075958 1177 raft_consensus.cc:357] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.076149 1177 raft_consensus.cc:383] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.076236 1177 raft_consensus.cc:738] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:27.076438 1177 consensus_queue.cc:260] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:27.077014 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 1a1e3af2883341e9b957f6ed748e604d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a1e3af2883341e9b957f6ed748e604d already in progress: creating tablet
I20250112 02:52:27.077489 1480 tablet.cc:1721] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:52:27.078359 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 35f59617b34046568dd83bbb7cdc42d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35f59617b34046568dd83bbb7cdc42d2 already in progress: creating tablet
W20250112 02:52:27.078843 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 5dd48d0e68014e1aa0c44620527baec9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5dd48d0e68014e1aa0c44620527baec9 already in progress: creating tablet
I20250112 02:52:27.081667 1177 ts_tablet_manager.cc:1428] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
W20250112 02:52:27.082139 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 1c639a2bcd4740e4be7b2b162004669a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c639a2bcd4740e4be7b2b162004669a already in progress: creating tablet
W20250112 02:52:27.075188 854 leader_election.cc:343] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:27.082708 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6913b0ac9ceb4dbc9cdf591f159134f7 already in progress: creating tablet
I20250112 02:52:27.082679 854 leader_election.cc:304] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:27.083230 1190 raft_consensus.cc:2743] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:27.034260 779 raft_consensus.cc:2463] T d3e41e307df64109a41b0c1c44d3bda6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
W20250112 02:52:27.083994 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f6803d0f553b4204b415b2734286bf4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6803d0f553b4204b415b2734286bf4a already in progress: creating tablet
I20250112 02:52:27.086391 991 leader_election.cc:304] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:27.087603 1177 tablet_bootstrap.cc:492] T f9186ac46ed441ee9308490e90da0e23 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:27.088806 825 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39987
W20250112 02:52:27.093770 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 64707ee432fb4e96a3bbf7252dfa1813 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64707ee432fb4e96a3bbf7252dfa1813 already in progress: creating tablet
W20250112 02:52:27.094316 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 9d5b3df88a0c45dba879b3d506710614 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d5b3df88a0c45dba879b3d506710614 already in progress: creating tablet
W20250112 02:52:27.095552 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 61593969d2234ce3a6089027b067d922 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61593969d2234ce3a6089027b067d922 already in progress: creating tablet
W20250112 02:52:27.096053 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 7a18132549374218893f35a4e4226b72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a18132549374218893f35a4e4226b72 already in progress: creating tablet
W20250112 02:52:27.096655 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f7cd0927c6a4401cbe69f55d4db9963a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7cd0927c6a4401cbe69f55d4db9963a already in progress: creating tablet
I20250112 02:52:27.097347 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet c7c530aa0aa64e47ae27d61f341d2bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.097427 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b599e983ec04f53a440d1ab828d12ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.097615 1177 tablet_bootstrap.cc:654] T f9186ac46ed441ee9308490e90da0e23 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:27.097889 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet b9199b0d2b924a79938a979df4196bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.098510 1030 tablet_service.cc:1467] Processing CreateTablet for tablet b9537fcd96eb41cc8f680075ec71137b (DEFAULT_TABLE table=default.Table_f22fde4d1df34a7b955de7c6eee20c3a [id=485dd281891f45e8b407b306572d6e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.098932 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9537fcd96eb41cc8f680075ec71137b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:27.099581 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 54033a747d5a49b0a1c16f138158f596 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54033a747d5a49b0a1c16f138158f596 already in progress: creating tablet
W20250112 02:52:27.101047 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet fb0aba0139cf479b80b32f3dfc936535 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb0aba0139cf479b80b32f3dfc936535 already in progress: creating tablet
W20250112 02:52:27.101562 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 8a12b225131b464494a9f02133d37aea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a12b225131b464494a9f02133d37aea already in progress: creating tablet
I20250112 02:52:27.101794 1171 tablet_replica.cc:331] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:27.101971 1171 raft_consensus.cc:2238] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:27.102100 1171 raft_consensus.cc:2267] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:27.102525 1171 ts_tablet_manager.cc:1905] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:27.105587 521 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 09d06e2e8c13464db8cf801a292bdc99 (table default.Table_0bdc5f49d4ff4d6fb1290bc2dc0f70d9 [id=e3bb489c53454222a79702b798c0c2f1]). Expected version 2 got 0
I20250112 02:52:27.111115 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_bac47d0dd2ab473babf687f5df1691c0" } new_table_name: "default.Table_8684181f6c4b40eb8948c3687408a1c4" modify_external_catalogs: true
I20250112 02:52:27.113008 1171 ts_tablet_manager.cc:1918] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:27.114305 524 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b6dc79f4d1194f81963a8094ed6b0d2e (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:27.116397 1171 log.cc:1198] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/d6ee0e79dbb34b1ba9dcd39d36aa2ae7
I20250112 02:52:27.117869 522 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 96e201d798c94e448a78e1c8adf41724 (table default.Table_4c59f5d61aab4b6184f0d9a4e21f965f [id=95cdab3e555643fc990b77b2ee8f058a]). Expected version 1 got 0
I20250112 02:52:27.119166 524 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b6dc79f4d1194f81963a8094ed6b0d2e (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:27.123600 1302 raft_consensus.cc:2798] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:27.125255 893 tablet_service.cc:1467] Processing CreateTablet for tablet c8a7281c0c09442ebd0cc13b8496c86e (DEFAULT_TABLE table=default.Table_105e5d1a8ef846ffbcb1bdbe01bbae55 [id=082744a588e64fe8bafe0bc442876fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.125736 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8a7281c0c09442ebd0cc13b8496c86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.127976 897 tablet_service.cc:1467] Processing CreateTablet for tablet b9d525dae23744a4adfc2f58cb9945ce (DEFAULT_TABLE table=default.Table_6f3879f2469d4655a6d4436362d0c3bb [id=bb0ba72d0ee747a3945283096d717008]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.128309 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9d525dae23744a4adfc2f58cb9945ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.128808 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_75db5f75b2874392b44dd580e0e52de9" } modify_external_catalogs: true
I20250112 02:52:27.129009 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_75db5f75b2874392b44dd580e0e52de9" } modify_external_catalogs: true
I20250112 02:52:27.124614 754 tablet_service.cc:1467] Processing CreateTablet for tablet c8a7281c0c09442ebd0cc13b8496c86e (DEFAULT_TABLE table=default.Table_105e5d1a8ef846ffbcb1bdbe01bbae55 [id=082744a588e64fe8bafe0bc442876fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.129933 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8a7281c0c09442ebd0cc13b8496c86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.130565 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_bee5a719ea4b44b78254bae566a0ba92" } modify_external_catalogs: true
I20250112 02:52:27.130692 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_bee5a719ea4b44b78254bae566a0ba92" } modify_external_catalogs: true
I20250112 02:52:27.131728 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 416fb6820cd54e118950c68d381f51b2 (DEFAULT_TABLE table=default.Table_f4d612cd6a524bc493147493c5a4a284 [id=4ac16bc62bbe4ed7a977d22c92ad580d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.132112 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 416fb6820cd54e118950c68d381f51b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.126273 751 tablet_service.cc:1467] Processing CreateTablet for tablet b9d525dae23744a4adfc2f58cb9945ce (DEFAULT_TABLE table=default.Table_6f3879f2469d4655a6d4436362d0c3bb [id=bb0ba72d0ee747a3945283096d717008]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.131732 1022 tablet_service.cc:1467] Processing CreateTablet for tablet b3c5a861a90840d0a33b780ec75bffd2 (DEFAULT_TABLE table=default.Table_c26869f6cbd540fe980dcf207b5c729d [id=cdd6a2097792484d84bca856f30d7ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.135838 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9d525dae23744a4adfc2f58cb9945ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.136108 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3c5a861a90840d0a33b780ec75bffd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.138844 1190 raft_consensus.cc:491] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:27.139029 1190 raft_consensus.cc:513] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.139824 750 tablet_service.cc:1467] Processing CreateTablet for tablet 223eb8b4d1024985844867d5641600d1 (DEFAULT_TABLE table=default.Table_e1d4a3104fc0481287acede4261941c6 [id=231e339ce95d48c99307e09b9de230c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.140164 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 223eb8b4d1024985844867d5641600d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.140262 891 tablet_service.cc:1467] Processing CreateTablet for tablet 223eb8b4d1024985844867d5641600d1 (DEFAULT_TABLE table=default.Table_e1d4a3104fc0481287acede4261941c6 [id=231e339ce95d48c99307e09b9de230c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.140646 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 223eb8b4d1024985844867d5641600d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.141196 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecfc7d8ba5e4406ebc61f37e6d500473" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:27.141678 854 leader_election.cc:343] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.141839 1190 leader_election.cc:290] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:27.141989 1190 raft_consensus.cc:491] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:27.142098 1190 raft_consensus.cc:513] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.143049 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1726d61cc36a4deb83c5d604e8daf4f0" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:27.143498 854 leader_election.cc:343] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.143800 753 tablet_service.cc:1467] Processing CreateTablet for tablet 2361bc6bd8164c2daa37be6b0845f4bd (DEFAULT_TABLE table=default.Table_ad5f01f6a5274cc483c8f051059d36c3 [id=8be34ef1e0c54f7fa38ec57fb432b3db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.144171 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2361bc6bd8164c2daa37be6b0845f4bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.144119 1020 tablet_service.cc:1467] Processing CreateTablet for tablet c8a7281c0c09442ebd0cc13b8496c86e (DEFAULT_TABLE table=default.Table_105e5d1a8ef846ffbcb1bdbe01bbae55 [id=082744a588e64fe8bafe0bc442876fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.144408 1190 leader_election.cc:290] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:27.144435 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8a7281c0c09442ebd0cc13b8496c86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.148453 888 tablet_service.cc:1467] Processing CreateTablet for tablet 2361bc6bd8164c2daa37be6b0845f4bd (DEFAULT_TABLE table=default.Table_ad5f01f6a5274cc483c8f051059d36c3 [id=8be34ef1e0c54f7fa38ec57fb432b3db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.148814 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2361bc6bd8164c2daa37be6b0845f4bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.148838 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1726d61cc36a4deb83c5d604e8daf4f0" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:27.148799 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_f8a40c27bebc483cbf62d5d0a4c7fb24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.149341 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8a40c27bebc483cbf62d5d0a4c7fb24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:27.149547 855 leader_election.cc:343] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.149672 855 leader_election.cc:304] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:27.150074 1457 raft_consensus.cc:2743] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:27.152086 896 tablet_service.cc:1467] Processing CreateTablet for tablet f234256b343f447dadfc3d1936a824ee (DEFAULT_TABLE table=default.Table_4603068c01544e2e965dca5177e36b40 [id=8502f21e9b61436093db1241cfbb5ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.152477 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f234256b343f447dadfc3d1936a824ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.153290 760 tablet_service.cc:1467] Processing CreateTablet for tablet f234256b343f447dadfc3d1936a824ee (DEFAULT_TABLE table=default.Table_4603068c01544e2e965dca5177e36b40 [id=8502f21e9b61436093db1241cfbb5ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.153641 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f234256b343f447dadfc3d1936a824ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.156513 757 tablet_service.cc:1467] Processing CreateTablet for tablet b8daf4ef3593410eae87672402292be6 (DEFAULT_TABLE table=default.Table_c0aaadb5e69e46e8983f4d289a44afe2 [id=08299ab225a94e37a67722107a5ff4ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.156576 749 tablet_service.cc:1467] Processing CreateTablet for tablet e5c10fad790046a88fbc8ebe0303c481 (DEFAULT_TABLE table=default.Table_16b036d8833a47a99577ab67579ccb5a [id=eb980088c4584ccf82bb585c2210894a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.156826 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8daf4ef3593410eae87672402292be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.156875 758 tablet_service.cc:1467] Processing CreateTablet for tablet d45fce86e02f46d4a57af718fa2a62c8 (DEFAULT_TABLE table=default.Table_34c47be51dd0466ca34b2fdc2bba8919 [id=370e66eb490a48b5927d073f94481de0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.157198 756 tablet_service.cc:1467] Processing CreateTablet for tablet e2f1f1a76e5f4308bf56dd647f02c6d8 (DEFAULT_TABLE table=default.Table_672d7452ad6d4c99ae9f2ae77b592f78 [id=33dad7ab4106404aaf33266ed1ea5635]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.159744 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d45fce86e02f46d4a57af718fa2a62c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.161306 892 tablet_service.cc:1467] Processing CreateTablet for tablet e5c10fad790046a88fbc8ebe0303c481 (DEFAULT_TABLE table=default.Table_16b036d8833a47a99577ab67579ccb5a [id=eb980088c4584ccf82bb585c2210894a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.161666 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5c10fad790046a88fbc8ebe0303c481. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.163890 886 tablet_service.cc:1467] Processing CreateTablet for tablet d45fce86e02f46d4a57af718fa2a62c8 (DEFAULT_TABLE table=default.Table_34c47be51dd0466ca34b2fdc2bba8919 [id=370e66eb490a48b5927d073f94481de0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.164180 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2f1f1a76e5f4308bf56dd647f02c6d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.164244 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d45fce86e02f46d4a57af718fa2a62c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.166662 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecfc7d8ba5e4406ebc61f37e6d500473" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:27.167380 754 tablet_service.cc:1467] Processing CreateTablet for tablet f0dc78d722ac4687bfb3dce94d6db7f0 (DEFAULT_TABLE table=default.Table_0e7ff0c4e04e4b23a8baf069f96ac465 [id=d98e9e3bcb104f48a5102dea02c73c95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.167541 895 tablet_service.cc:1467] Processing CreateTablet for tablet e2f1f1a76e5f4308bf56dd647f02c6d8 (DEFAULT_TABLE table=default.Table_672d7452ad6d4c99ae9f2ae77b592f78 [id=33dad7ab4106404aaf33266ed1ea5635]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.167662 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5c10fad790046a88fbc8ebe0303c481. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.167820 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0dc78d722ac4687bfb3dce94d6db7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.167886 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2f1f1a76e5f4308bf56dd647f02c6d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.168432 898 tablet_service.cc:1467] Processing CreateTablet for tablet b8daf4ef3593410eae87672402292be6 (DEFAULT_TABLE table=default.Table_c0aaadb5e69e46e8983f4d289a44afe2 [id=08299ab225a94e37a67722107a5ff4ce]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:27.168946 855 leader_election.cc:343] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.169081 855 leader_election.cc:304] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:27.170127 1457 raft_consensus.cc:491] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:27.170123 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8daf4ef3593410eae87672402292be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.178669 1025 tablet_service.cc:1467] Processing CreateTablet for tablet b9d525dae23744a4adfc2f58cb9945ce (DEFAULT_TABLE table=default.Table_6f3879f2469d4655a6d4436362d0c3bb [id=bb0ba72d0ee747a3945283096d717008]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.179119 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9d525dae23744a4adfc2f58cb9945ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.179286 1171 ts_tablet_manager.cc:1939] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:27.180235 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 (table default.Table_c360a95859e5448f9af5f90629a898e4 [id=08b0c7e4e0c54ec4a215a390e83a1cf6]) successfully deleted
I20250112 02:52:27.185532 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 2361bc6bd8164c2daa37be6b0845f4bd (DEFAULT_TABLE table=default.Table_ad5f01f6a5274cc483c8f051059d36c3 [id=8be34ef1e0c54f7fa38ec57fb432b3db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.185914 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2361bc6bd8164c2daa37be6b0845f4bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.186774 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b5f80ed198491780cbe17e6cd3fa15" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:27.187001 1026 tablet_service.cc:1467] Processing CreateTablet for tablet f234256b343f447dadfc3d1936a824ee (DEFAULT_TABLE table=default.Table_4603068c01544e2e965dca5177e36b40 [id=8502f21e9b61436093db1241cfbb5ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.187309 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f234256b343f447dadfc3d1936a824ee. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:27.190533 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1b184806b4884cd3b0a002905a1e57af consensus state has no leader
I20250112 02:52:27.191344 1171 tablet_replica.cc:331] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:27.191334 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_72e8ed041bf340c383458fd7c96807ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:27.191545 1171 raft_consensus.cc:2238] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:27.191721 1171 raft_consensus.cc:2267] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:27.191844 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72e8ed041bf340c383458fd7c96807ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.192224 1171 ts_tablet_manager.cc:1905] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:27.193516 1020 tablet_service.cc:1467] Processing CreateTablet for tablet e5c10fad790046a88fbc8ebe0303c481 (DEFAULT_TABLE table=default.Table_16b036d8833a47a99577ab67579ccb5a [id=eb980088c4584ccf82bb585c2210894a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.193660 1021 tablet_service.cc:1467] Processing CreateTablet for tablet b8daf4ef3593410eae87672402292be6 (DEFAULT_TABLE table=default.Table_c0aaadb5e69e46e8983f4d289a44afe2 [id=08299ab225a94e37a67722107a5ff4ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.193938 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5c10fad790046a88fbc8ebe0303c481. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.194155 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8daf4ef3593410eae87672402292be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.199890 1033 tablet_service.cc:1467] Processing CreateTablet for tablet e2f1f1a76e5f4308bf56dd647f02c6d8 (DEFAULT_TABLE table=default.Table_672d7452ad6d4c99ae9f2ae77b592f78 [id=33dad7ab4106404aaf33266ed1ea5635]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:27.200450 714 leader_election.cc:343] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.200591 714 leader_election.cc:304] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
W20250112 02:52:27.200816 714 raft_consensus.cc:2681] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 02:52:27.201529 1100 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39987
I20250112 02:52:27.202183 1488 tablet.cc:1721] T 96e201d798c94e448a78e1c8adf41724 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:27.203629 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 1a8a178bafcb457f9b0e9587df560493
I20250112 02:52:27.204344 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_1d107b48a09e41d6bb3401b01a21ee78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:27.204600 1026 tablet_service.cc:1467] Processing CreateTablet for tablet d45fce86e02f46d4a57af718fa2a62c8 (DEFAULT_TABLE table=default.Table_34c47be51dd0466ca34b2fdc2bba8919 [id=370e66eb490a48b5927d073f94481de0]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:27.204839 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d107b48a09e41d6bb3401b01a21ee78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.204900 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2f1f1a76e5f4308bf56dd647f02c6d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.205580 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 223eb8b4d1024985844867d5641600d1 (DEFAULT_TABLE table=default.Table_e1d4a3104fc0481287acede4261941c6 [id=231e339ce95d48c99307e09b9de230c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.206301 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet a85300a9aa1c4c50a3957a004bea7933
I20250112 02:52:27.206915 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_9c3a68b2aa9a4edcb80525136f7bfa96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.207350 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c3a68b2aa9a4edcb80525136f7bfa96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.207883 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet ac716c6da01941a989754aac1ef430d5
I20250112 02:52:27.208438 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_ed3c26ab0711404581dcd62adb343dde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:27.208657 1171 ts_tablet_manager.cc:1918] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 02:52:27.208882 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed3c26ab0711404581dcd62adb343dde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.209012 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 223eb8b4d1024985844867d5641600d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.209854 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d45fce86e02f46d4a57af718fa2a62c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.210393 1171 log.cc:1198] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/71b5f80ed198491780cbe17e6cd3fa15
I20250112 02:52:27.210512 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 6cb76746b3a846c3aba531c04bf63966
I20250112 02:52:27.210947 1171 ts_tablet_manager.cc:1939] T 71b5f80ed198491780cbe17e6cd3fa15 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:27.211274 1030 tablet_service.cc:1467] Processing CreateTablet for tablet a12c5d58e6c246c983e8a533906ce77d (DEFAULT_TABLE table=default.Table_f7bab317410d408aa363857a99be6e3c [id=d0be0fa53a79498e9160bfaa7d88837b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.211688 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a12c5d58e6c246c983e8a533906ce77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.211794 1171 tablet_replica.cc:331] T d3e41e307df64109a41b0c1c44d3bda6 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:27.212144 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet c8a7281c0c09442ebd0cc13b8496c86e
I20250112 02:52:27.212852 1024 tablet_service.cc:1467] Processing CreateTablet for tablet fe614eaeb793473d9b2da206b7345562 (DEFAULT_TABLE table=default.Table_b2b3c738a24f451eac39fb1a6399f37a [id=7c6c2713869e476cb23d157a285a5afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.213306 1171 raft_consensus.cc:2238] T d3e41e307df64109a41b0c1c44d3bda6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:27.213812 1171 raft_consensus.cc:2267] T d3e41e307df64109a41b0c1c44d3bda6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:27.214536 1171 ts_tablet_manager.cc:1905] T d3e41e307df64109a41b0c1c44d3bda6 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:27.215648 1302 raft_consensus.cc:695] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:27.215972 1302 consensus_queue.cc:237] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.216281 751 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a7281c0c09442ebd0cc13b8496c86e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:27.216281 749 tablet_service.cc:1514] Processing DeleteTablet for tablet ac716c6da01941a989754aac1ef430d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:27.216732 518 catalog_manager.cc:6071] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: default.Table_4c59f5d61aab4b6184f0d9a4e21f965f [id=95cdab3e555643fc990b77b2ee8f058a] alter complete (version 1)
I20250112 02:52:27.217202 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe614eaeb793473d9b2da206b7345562. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.219975 1190 raft_consensus.cc:2743] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:27.220103 756 tablet_service.cc:1514] Processing DeleteTablet for tablet a85300a9aa1c4c50a3957a004bea7933 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:27.221590 1457 raft_consensus.cc:513] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.222007 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb76746b3a846c3aba531c04bf63966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:27.224362 897 tablet_service.cc:1467] Processing CreateTablet for tablet f0dc78d722ac4687bfb3dce94d6db7f0 (DEFAULT_TABLE table=default.Table_0e7ff0c4e04e4b23a8baf069f96ac465 [id=d98e9e3bcb104f48a5102dea02c73c95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.232332 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_8dd5b6fa78f64fe78de385801d15fafa" } new_table_name: "default.Table_743b3f7029534564b35c0c137b35fd75" modify_external_catalogs: true
I20250112 02:52:27.233415 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_2740664d25a8451b87667c0eaf8b4d1b" } new_table_name: "default.Table_0dec5de812ab4ea48f1b7a8f4cb0cc4f" modify_external_catalogs: true
I20250112 02:52:27.234452 518 master_service.cc:946] ReplaceTablet: received request to replace tablet 015412a5fa304453bd8d1be1c3cd7a46 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:27.235322 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a8a178bafcb457f9b0e9587df560493 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.235411 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a8a178bafcb457f9b0e9587df560493 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:27.236253 1458 raft_consensus.cc:491] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:27.237494 1171 ts_tablet_manager.cc:1918] T d3e41e307df64109a41b0c1c44d3bda6 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:27.237671 1171 log.cc:1198] T d3e41e307df64109a41b0c1c44d3bda6 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/d3e41e307df64109a41b0c1c44d3bda6
I20250112 02:52:27.238061 1171 ts_tablet_manager.cc:1939] T d3e41e307df64109a41b0c1c44d3bda6 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:27.238653 1457 leader_election.cc:290] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:27.239012 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet d3e41e307df64109a41b0c1c44d3bda6 (table default.Table_01a0b2e48fcd4a769a3d442390622c1e [id=4d04607bb9004f3cb1871c43d0eb47ce]) successfully deleted
I20250112 02:52:27.239214 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0dc78d722ac4687bfb3dce94d6db7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.240129 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54033a747d5a49b0a1c16f138158f596" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:27.240360 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_fb003efcf38c4d18897952ba1cfe5ca8" } new_table_name: "default.Table_4934f6022f2441eda88751236a5798be" modify_external_catalogs: true
I20250112 02:52:27.240465 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54033a747d5a49b0a1c16f138158f596" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:27.241420 1458 raft_consensus.cc:513] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.241811 896 tablet_service.cc:1514] Processing DeleteTablet for tablet a85300a9aa1c4c50a3957a004bea7933 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:27.242733 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b9199b0d2b924a79938a979df4196bb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9199b0d2b924a79938a979df4196bb4 already in progress: creating tablet
W20250112 02:52:27.243745 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet c7c530aa0aa64e47ae27d61f341d2bd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7c530aa0aa64e47ae27d61f341d2bd9 already in progress: creating tablet
W20250112 02:52:27.244272 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 4b599e983ec04f53a440d1ab828d12ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b599e983ec04f53a440d1ab828d12ab already in progress: creating tablet
I20250112 02:52:27.244536 1171 tablet_replica.cc:331] T fb4c004397ee480d897e8b125d248829 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:27.244701 1171 raft_consensus.cc:2238] T fb4c004397ee480d897e8b125d248829 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:27.244846 1171 raft_consensus.cc:2267] T fb4c004397ee480d897e8b125d248829 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:27.245285 1171 ts_tablet_manager.cc:1905] T fb4c004397ee480d897e8b125d248829 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:27.245414 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb76746b3a846c3aba531c04bf63966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.241160 893 tablet_service.cc:1514] Processing DeleteTablet for tablet ac716c6da01941a989754aac1ef430d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:27.246093 854 leader_election.cc:343] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.248371 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_dd5ce8476cb84204b0351fd07cab2b00" } new_table_name: "default.Table_ed757d89486c45058437db3268063639" modify_external_catalogs: true
I20250112 02:52:27.258502 1171 ts_tablet_manager.cc:1918] T fb4c004397ee480d897e8b125d248829 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:27.258711 1171 log.cc:1198] T fb4c004397ee480d897e8b125d248829 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/fb4c004397ee480d897e8b125d248829
I20250112 02:52:27.259227 1171 ts_tablet_manager.cc:1939] T fb4c004397ee480d897e8b125d248829 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:27.260504 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet fb4c004397ee480d897e8b125d248829 (table default.Table_091c87de29164fc784158659eeaf545a [id=b8c138bea585437c83ee0ed94c008284]) successfully deleted
I20250112 02:52:27.260957 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 1726d61cc36a4deb83c5d604e8daf4f0 with delete_type TABLET_DATA_DELETED (replaced by tablet b2ccb88108f24e36ba4b7ebf36398950) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.262029 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_7077f9a718664d8b915ee4003502306a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.262460 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7077f9a718664d8b915ee4003502306a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.264360 1171 tablet_replica.cc:331] T 9427951452104a7db765437493b21e90 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:27.264609 1171 raft_consensus.cc:2238] T 9427951452104a7db765437493b21e90 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:27.264817 1171 raft_consensus.cc:2267] T 9427951452104a7db765437493b21e90 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:27.265522 1171 ts_tablet_manager.cc:1905] T 9427951452104a7db765437493b21e90 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:27.266373 855 leader_election.cc:343] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.267582 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97b31ebe035c4552baaffc3133ec73c1" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:27.267681 1458 leader_election.cc:290] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:27.268754 854 leader_election.cc:343] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.270310 855 leader_election.cc:304] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:27.271317 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97b31ebe035c4552baaffc3133ec73c1" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:27.272747 1171 ts_tablet_manager.cc:1918] T 9427951452104a7db765437493b21e90 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:27.272886 1458 raft_consensus.cc:2743] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:27.272954 1171 log.cc:1198] T 9427951452104a7db765437493b21e90 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/9427951452104a7db765437493b21e90
I20250112 02:52:27.273367 1171 ts_tablet_manager.cc:1939] T 9427951452104a7db765437493b21e90 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:27.274343 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 9427951452104a7db765437493b21e90 (table default.Table_d38ef91002ba4653916c476d1df3a9c6 [id=fd0dc6c5d5d14251a3e6b52e3d66ed0e]) successfully deleted
W20250112 02:52:27.274950 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet c8a7281c0c09442ebd0cc13b8496c86e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8a7281c0c09442ebd0cc13b8496c86e already in progress: creating tablet
W20250112 02:52:27.275658 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet ac716c6da01941a989754aac1ef430d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac716c6da01941a989754aac1ef430d5 already in progress: creating tablet
W20250112 02:52:27.276279 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet a85300a9aa1c4c50a3957a004bea7933 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a85300a9aa1c4c50a3957a004bea7933 already in progress: creating tablet
W20250112 02:52:27.276739 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 6cb76746b3a846c3aba531c04bf63966 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6cb76746b3a846c3aba531c04bf63966 already in progress: creating tablet
W20250112 02:52:27.277227 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 1a8a178bafcb457f9b0e9587df560493 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a8a178bafcb457f9b0e9587df560493 already in progress: creating tablet
W20250112 02:52:27.287503 855 leader_election.cc:343] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.287715 855 leader_election.cc:304] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:27.288210 1458 raft_consensus.cc:2743] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:27.288458 893 tablet_service.cc:1514] Processing DeleteTablet for tablet afa2f8aabe5e42ccbc15b193fdc03563 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.288729 896 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a7281c0c09442ebd0cc13b8496c86e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.296420 1458 raft_consensus.cc:491] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:27.296563 1458 raft_consensus.cc:513] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.297117 1458 leader_election.cc:290] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:27.297724 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d8ea2ef18a46508371b4789ab6f4fa" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:27.297739 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d8ea2ef18a46508371b4789ab6f4fa" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:27.298233 854 leader_election.cc:343] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:27.298493 855 leader_election.cc:343] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.298614 855 leader_election.cc:304] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:27.298940 1458 raft_consensus.cc:2743] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:27.311867 1175 ts_tablet_manager.cc:1918] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:27.312141 1175 log.cc:1198] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/c2595782d66e4e4b890c94221b196daa
I20250112 02:52:27.312165 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_150748c55de444749d147c320c6d8c0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:27.312705 1175 ts_tablet_manager.cc:1939] T c2595782d66e4e4b890c94221b196daa P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
W20250112 02:52:27.312814 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_150748c55de444749d147c320c6d8c0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.313691 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 211) took 1552 ms. Trace:
I20250112 02:52:27.314474 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_ec85ae9d8f3c49699f712c3104096f2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.314944 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec85ae9d8f3c49699f712c3104096f2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.315901 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_2750897a0a504c2ca90590075d4cf54e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.316402 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2750897a0a504c2ca90590075d4cf54e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.317003 1177 tablet_bootstrap.cc:492] T f9186ac46ed441ee9308490e90da0e23 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:27.317148 1177 ts_tablet_manager.cc:1397] T f9186ac46ed441ee9308490e90da0e23 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.230s user 0.033s sys 0.043s
I20250112 02:52:27.317940 1177 raft_consensus.cc:357] T f9186ac46ed441ee9308490e90da0e23 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:27.318049 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 003a3b4705e943c6a900762ec11ddd47 has no consensus state
I20250112 02:52:27.318133 1177 raft_consensus.cc:383] T f9186ac46ed441ee9308490e90da0e23 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.318199 1177 raft_consensus.cc:738] T f9186ac46ed441ee9308490e90da0e23 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
W20250112 02:52:27.318462 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d0c3191973a477e949fbf9dfcca963f consensus state has no leader
I20250112 02:52:27.318434 1177 consensus_queue.cc:260] T f9186ac46ed441ee9308490e90da0e23 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:27.318804 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c03b5e612d64314b210c11686fdfa2a consensus state has no leader
W20250112 02:52:27.319140 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d25815a5df94e099e7481c48c3da793 consensus state has no leader
I20250112 02:52:27.313797 1175 rpcz_store.cc:276] 0112 02:52:25.761034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:25.761311 (+ 277us) service_pool.cc:225] Handling call
0112 02:52:26.973730 (+1212419us) ts_tablet_manager.cc:1145] Deleting tablet c2595782d66e4e4b890c94221b196daa
0112 02:52:26.973738 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:26.999291 (+ 25553us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:27.311726 (+312435us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:27.313630 (+ 1904us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1211635}
I20250112 02:52:27.322422 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet c2595782d66e4e4b890c94221b196daa (table default.Table_96a35d4e94284a49ba1a88d7d0acd3c1 [id=8559c454b17b4a00a60689a6a9b19f7d]) successfully deleted
I20250112 02:52:27.322688 1458 raft_consensus.cc:491] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:27.322856 1458 raft_consensus.cc:513] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.323439 1458 leader_election.cc:290] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:27.323673 754 tablet_service.cc:1514] Processing DeleteTablet for tablet f9186ac46ed441ee9308490e90da0e23 with delete_type TABLET_DATA_DELETED (replaced by tablet 04f135b226774285a077e5806131d91b) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:27.324118 1177 ts_tablet_manager.cc:1428] T f9186ac46ed441ee9308490e90da0e23 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
W20250112 02:52:27.324508 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f9186ac46ed441ee9308490e90da0e23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9186ac46ed441ee9308490e90da0e23 already in progress: creating tablet
I20250112 02:52:27.325110 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa2f8aabe5e42ccbc15b193fdc03563" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:27.325619 855 leader_election.cc:343] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.326098 1175 tablet_replica.cc:331] T e9fc9a4f1e8944abb76dad6c2ad53d35 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:27.326802 754 tablet_service.cc:1514] Processing DeleteTablet for tablet f9186ac46ed441ee9308490e90da0e23 with delete_type TABLET_DATA_DELETED (replaced by tablet 04f135b226774285a077e5806131d91b) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:27.327030 1171 tablet_replica.cc:331] T f9186ac46ed441ee9308490e90da0e23 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:27.327101 1177 tablet_bootstrap.cc:492] T a47cdc208fbc431b8326364e84432cf5 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:27.327209 1171 raft_consensus.cc:2238] T f9186ac46ed441ee9308490e90da0e23 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:27.327206 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 015412a5fa304453bd8d1be1c3cd7a46
I20250112 02:52:27.327370 1171 raft_consensus.cc:2267] T f9186ac46ed441ee9308490e90da0e23 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:27.327724 518 catalog_manager.cc:6596] ReplaceTablet: tablet 015412a5fa304453bd8d1be1c3cd7a46 deleted and replaced by tablet f924459d1150472aafb325a11825db23
I20250112 02:52:27.329555 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_24cecfd421cd4047972cc0a6255ef0c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.330174 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24cecfd421cd4047972cc0a6255ef0c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.331305 1177 tablet_bootstrap.cc:654] T a47cdc208fbc431b8326364e84432cf5 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:27.332731 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 015412a5fa304453bd8d1be1c3cd7a46 with delete_type TABLET_DATA_DELETED (replaced by tablet f924459d1150472aafb325a11825db23) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:27.336107 1171 ts_tablet_manager.cc:1905] T f9186ac46ed441ee9308490e90da0e23 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:27.340683 1172 log.cc:1198] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/235e9dd0964c4ea6af7d05ef58acae0d
I20250112 02:52:27.347602 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 015412a5fa304453bd8d1be1c3cd7a46 with delete_type TABLET_DATA_DELETED (replaced by tablet f924459d1150472aafb325a11825db23) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.348906 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 5d0c3191973a477e949fbf9dfcca963f (DEFAULT_TABLE table=default.Table_2740664d25a8451b87667c0eaf8b4d1b [id=2f43ebd749304c0287769224ac6b12c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.349236 1172 ts_tablet_manager.cc:1939] T 235e9dd0964c4ea6af7d05ef58acae0d P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
W20250112 02:52:27.350782 512 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:42341, user_credentials={real_user=slave}} blocked reactor thread for 45351.4us
I20250112 02:52:27.358741 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d0c3191973a477e949fbf9dfcca963f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.358817 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa2f8aabe5e42ccbc15b193fdc03563" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:27.359421 854 leader_election.cc:343] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.359612 854 leader_election.cc:304] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:27.359719 518 master_service.cc:946] ReplaceTablet: received request to replace tablet 1b184806b4884cd3b0a002905a1e57af from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:27.360062 1458 raft_consensus.cc:2743] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:27.360219 1022 tablet_service.cc:1467] Processing CreateTablet for tablet f0dc78d722ac4687bfb3dce94d6db7f0 (DEFAULT_TABLE table=default.Table_0e7ff0c4e04e4b23a8baf069f96ac465 [id=d98e9e3bcb104f48a5102dea02c73c95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.360630 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0dc78d722ac4687bfb3dce94d6db7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.370124 1172 tablet_replica.cc:331] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:27.370343 1172 raft_consensus.cc:2238] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:27.371490 1172 pending_rounds.cc:62] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6: Trying to abort 1 pending ops.
I20250112 02:52:27.371639 1172 pending_rounds.cc:69] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319822199115776 op_type: NO_OP noop_request { }
I20250112 02:52:27.371855 525 master_service.cc:946] ReplaceTablet: received request to replace tablet f6803d0f553b4204b415b2734286bf4a from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:27.372665 1172 raft_consensus.cc:2883] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:27.372859 1172 raft_consensus.cc:2267] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:27.377059 1171 ts_tablet_manager.cc:1918] T f9186ac46ed441ee9308490e90da0e23 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:27.377408 1171 log.cc:1198] T f9186ac46ed441ee9308490e90da0e23 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/f9186ac46ed441ee9308490e90da0e23
I20250112 02:52:27.378161 1171 ts_tablet_manager.cc:1939] T f9186ac46ed441ee9308490e90da0e23 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:27.382090 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 235e9dd0964c4ea6af7d05ef58acae0d (table default.Table_ae0485ac89ea47e7a8c19a07c5a63730 [id=e659e0cc87684ca79fc3b8241116742a]) successfully deleted
I20250112 02:52:27.391991 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_b8256597ba354774ace25add22300525" } new_table_name: "default.Table_e3c5ba009c9f4301a3f99d51a8dcf687" modify_external_catalogs: true
I20250112 02:52:27.396261 1175 raft_consensus.cc:2238] T e9fc9a4f1e8944abb76dad6c2ad53d35 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:27.396580 1175 raft_consensus.cc:2267] T e9fc9a4f1e8944abb76dad6c2ad53d35 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:27.397403 1173 tablet_bootstrap.cc:492] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:27.397696 1181 tablet_bootstrap.cc:492] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:27.397735 1173 ts_tablet_manager.cc:1397] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.544s user 0.003s sys 0.003s
I20250112 02:52:27.397845 1181 ts_tablet_manager.cc:1397] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.327s user 0.019s sys 0.072s
W20250112 02:52:27.398140 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 73136be25c79491784d3ea71aa9b27f2 consensus state has no leader
I20250112 02:52:27.398607 1181 raft_consensus.cc:357] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.398811 1181 raft_consensus.cc:383] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.398912 1181 raft_consensus.cc:738] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:27.398836 1173 raft_consensus.cc:357] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.399247 1173 raft_consensus.cc:383] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.399160 1181 consensus_queue.cc:260] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.399693 1181 ts_tablet_manager.cc:1428] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:27.400054 1181 tablet_bootstrap.cc:492] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:27.402333 1181 tablet_bootstrap.cc:654] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:27.405052 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39987
I20250112 02:52:27.406195 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c639a2bcd4740e4be7b2b162004669a) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.407914 1175 ts_tablet_manager.cc:1905] T e9fc9a4f1e8944abb76dad6c2ad53d35 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:27.413184 1173 raft_consensus.cc:738] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:27.414822 1173 consensus_queue.cc:260] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.417114 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_d1e5cef5fa3e4d90bada176f80bd4180" } modify_external_catalogs: true
I20250112 02:52:27.417327 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_d1e5cef5fa3e4d90bada176f80bd4180" } modify_external_catalogs: true
I20250112 02:52:27.417956 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet f9186ac46ed441ee9308490e90da0e23 (table default.Table_d1e5cef5fa3e4d90bada176f80bd4180 [id=f93b5bb22ec1424ea047383ed9713618]) successfully deleted
I20250112 02:52:27.418119 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_16b036d8833a47a99577ab67579ccb5a" } modify_external_catalogs: true
I20250112 02:52:27.418139 1173 ts_tablet_manager.cc:1428] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20250112 02:52:27.418292 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_16b036d8833a47a99577ab67579ccb5a" } modify_external_catalogs: true
I20250112 02:52:27.417963 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_af61980bca5e4995a77da652dcfb3f0d" } modify_external_catalogs: true
I20250112 02:52:27.418974 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_af61980bca5e4995a77da652dcfb3f0d" } modify_external_catalogs: true
I20250112 02:52:27.419055 1173 tablet_bootstrap.cc:492] T 49650a419dfd49d5a1547770d3f2fe54 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:27.419564 521 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_9d6d135049de4ecab068bd8ab4d9fad4" } modify_external_catalogs: true
I20250112 02:52:27.419703 521 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_9d6d135049de4ecab068bd8ab4d9fad4" } modify_external_catalogs: true
I20250112 02:52:27.421083 1181 tablet_bootstrap.cc:492] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:27.421258 1181 ts_tablet_manager.cc:1397] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.007s
I20250112 02:52:27.421912 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_c90865422da541e8a8665d0ba0c3fdb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:27.422003 1181 raft_consensus.cc:357] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.422179 1181 raft_consensus.cc:383] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.422246 1181 raft_consensus.cc:738] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
W20250112 02:52:27.422416 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c90865422da541e8a8665d0ba0c3fdb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.422444 1181 consensus_queue.cc:260] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.423113 1181 ts_tablet_manager.cc:1428] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:27.423451 1181 tablet_bootstrap.cc:492] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:27.423975 526 master_service.cc:946] ReplaceTablet: received request to replace tablet 6b521522c4784d2ab4056268823f1099 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:27.425750 1181 tablet_bootstrap.cc:654] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:27.427848 1294 consensus_queue.cc:1035] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:27.432333 896 tablet_service.cc:1467] Processing CreateTablet for tablet e4d42f1611de49d6b6e5aaa9be907054 (DEFAULT_TABLE table=default.Table_7926c082d6c04315811ea3b4e1579e8c [id=c01b653cd9fe45b2af721eb97936c5d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.432818 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d42f1611de49d6b6e5aaa9be907054. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.435524 1181 tablet_bootstrap.cc:492] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:27.435644 1181 ts_tablet_manager.cc:1397] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.006s
I20250112 02:52:27.436368 1181 raft_consensus.cc:357] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.436538 1181 raft_consensus.cc:383] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.436625 1181 raft_consensus.cc:738] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:27.436832 1181 consensus_queue.cc:260] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.437366 1181 ts_tablet_manager.cc:1428] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:27.437319 893 tablet_service.cc:1467] Processing CreateTablet for tablet c4a3486ef011492c8a1df8d6278adedf (DEFAULT_TABLE table=default.Table_33b6530b130041f0a2a4a7240e92ff15 [id=1d614fd3947a43bcac5a3d40cad6d80b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.437590 1181 tablet_bootstrap.cc:492] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:27.437669 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4a3486ef011492c8a1df8d6278adedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.438359 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb76746b3a846c3aba531c04bf63966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.438526 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet a85300a9aa1c4c50a3957a004bea7933 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.439713 897 tablet_service.cc:1467] Processing CreateTablet for tablet f05c8426ea3d4fe89fbe6c88c688019f (DEFAULT_TABLE table=default.Table_952b91d2b0b8435886e555f12629dc80 [id=ed2c162c64f74c9f889789b3abe7c041]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.440021 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f05c8426ea3d4fe89fbe6c88c688019f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.441186 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet ac716c6da01941a989754aac1ef430d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.441694 1181 tablet_bootstrap.cc:654] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:27.443689 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a8a178bafcb457f9b0e9587df560493 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:50844
W20250112 02:52:27.444604 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 015412a5fa304453bd8d1be1c3cd7a46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 015412a5fa304453bd8d1be1c3cd7a46 already in progress: creating tablet
I20250112 02:52:27.445102 754 tablet_service.cc:1467] Processing CreateTablet for tablet e4d42f1611de49d6b6e5aaa9be907054 (DEFAULT_TABLE table=default.Table_7926c082d6c04315811ea3b4e1579e8c [id=c01b653cd9fe45b2af721eb97936c5d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.445497 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d42f1611de49d6b6e5aaa9be907054. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.447594 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet df6ccd04880442d0990c6cb0d27956bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.447728 757 tablet_service.cc:1467] Processing CreateTablet for tablet c4a3486ef011492c8a1df8d6278adedf (DEFAULT_TABLE table=default.Table_33b6530b130041f0a2a4a7240e92ff15 [id=1d614fd3947a43bcac5a3d40cad6d80b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.448055 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4a3486ef011492c8a1df8d6278adedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.451686 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 015412a5fa304453bd8d1be1c3cd7a46 with delete_type TABLET_DATA_DELETED (replaced by tablet f924459d1150472aafb325a11825db23) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.451802 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a7281c0c09442ebd0cc13b8496c86e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.451767 758 tablet_service.cc:1467] Processing CreateTablet for tablet f05c8426ea3d4fe89fbe6c88c688019f (DEFAULT_TABLE table=default.Table_952b91d2b0b8435886e555f12629dc80 [id=ed2c162c64f74c9f889789b3abe7c041]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.452087 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f05c8426ea3d4fe89fbe6c88c688019f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.455127 1023 tablet_service.cc:1467] Processing CreateTablet for tablet e4d42f1611de49d6b6e5aaa9be907054 (DEFAULT_TABLE table=default.Table_7926c082d6c04315811ea3b4e1579e8c [id=c01b653cd9fe45b2af721eb97936c5d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.455453 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d42f1611de49d6b6e5aaa9be907054. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.456251 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 3567820f0d4349799b681fdfff0ebad8 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a18132549374218893f35a4e4226b72) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.463182 756 tablet_service.cc:1467] Processing CreateTablet for tablet 856fa83dedfd4b049e3e0d91d7ac5e40 (DEFAULT_TABLE table=default.Table_4657847b84384e628194a8ff43e183a1 [id=85db347bdf4a4ff583062f3dddd20d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.463578 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 856fa83dedfd4b049e3e0d91d7ac5e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.464293 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b521522c4784d2ab4056268823f1099 with delete_type TABLET_DATA_DELETED (replaced by tablet ab2e31a65294407bb62b13e92898a7d2) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.467150 1458 raft_consensus.cc:491] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:27.467316 1458 raft_consensus.cc:513] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.467878 1458 leader_election.cc:290] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:27.468457 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6803d0f553b4204b415b2734286bf4a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:27.469112 855 leader_election.cc:343] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.474054 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_33b6530b130041f0a2a4a7240e92ff15" } modify_external_catalogs: true
I20250112 02:52:27.474246 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_33b6530b130041f0a2a4a7240e92ff15" } modify_external_catalogs: true
I20250112 02:52:27.475997 1034 tablet_service.cc:1467] Processing CreateTablet for tablet c4a3486ef011492c8a1df8d6278adedf (DEFAULT_TABLE table=default.Table_33b6530b130041f0a2a4a7240e92ff15 [id=1d614fd3947a43bcac5a3d40cad6d80b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.476375 895 tablet_service.cc:1467] Processing CreateTablet for tablet 856fa83dedfd4b049e3e0d91d7ac5e40 (DEFAULT_TABLE table=default.Table_4657847b84384e628194a8ff43e183a1 [id=85db347bdf4a4ff583062f3dddd20d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.476537 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4a3486ef011492c8a1df8d6278adedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.476686 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 856fa83dedfd4b049e3e0d91d7ac5e40. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:27.462407 1489 consensus_queue.cc:1159] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f1e775296eb746db8b318f17d26820e6" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250112 02:52:27.484432 1020 tablet_service.cc:1467] Processing CreateTablet for tablet f05c8426ea3d4fe89fbe6c88c688019f (DEFAULT_TABLE table=default.Table_952b91d2b0b8435886e555f12629dc80 [id=ed2c162c64f74c9f889789b3abe7c041]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.486156 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 5a8e1b96accd459eaed221ded4a453b6
I20250112 02:52:27.487362 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 1b184806b4884cd3b0a002905a1e57af
I20250112 02:52:27.487802 518 catalog_manager.cc:6596] ReplaceTablet: tablet 1b184806b4884cd3b0a002905a1e57af deleted and replaced by tablet 47a0d939fdb94590a86389f1f4693e9a
I20250112 02:52:27.487932 1172 ts_tablet_manager.cc:1905] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:27.488727 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8e1b96accd459eaed221ded4a453b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.488912 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b184806b4884cd3b0a002905a1e57af with delete_type TABLET_DATA_DELETED (replaced by tablet 47a0d939fdb94590a86389f1f4693e9a) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.489907 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet e5c10fad790046a88fbc8ebe0303c481
I20250112 02:52:27.491096 886 tablet_service.cc:1514] Processing DeleteTablet for tablet e5c10fad790046a88fbc8ebe0303c481 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.491845 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b184806b4884cd3b0a002905a1e57af with delete_type TABLET_DATA_DELETED (replaced by tablet 47a0d939fdb94590a86389f1f4693e9a) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:27.492065 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 7a34d03a80ae402db6607261a5c40b60
I20250112 02:52:27.492146 751 tablet_service.cc:1514] Processing DeleteTablet for tablet e5c10fad790046a88fbc8ebe0303c481 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:27.490798 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 856fa83dedfd4b049e3e0d91d7ac5e40 (DEFAULT_TABLE table=default.Table_4657847b84384e628194a8ff43e183a1 [id=85db347bdf4a4ff583062f3dddd20d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.492699 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6803d0f553b4204b415b2734286bf4a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:27.493460 854 leader_election.cc:343] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.493623 854 leader_election.cc:304] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:27.493772 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a34d03a80ae402db6607261a5c40b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:27.493481 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a34d03a80ae402db6607261a5c40b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.495535 1458 raft_consensus.cc:2743] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:27.499032 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8e1b96accd459eaed221ded4a453b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.499737 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f05c8426ea3d4fe89fbe6c88c688019f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.500792 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet e5c10fad790046a88fbc8ebe0303c481 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.501917 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b184806b4884cd3b0a002905a1e57af with delete_type TABLET_DATA_DELETED (replaced by tablet 47a0d939fdb94590a86389f1f4693e9a) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.502220 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 856fa83dedfd4b049e3e0d91d7ac5e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.506155 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a34d03a80ae402db6607261a5c40b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.511348 1173 tablet_bootstrap.cc:654] T 49650a419dfd49d5a1547770d3f2fe54 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:27.520104 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 04f135b226774285a077e5806131d91b
I20250112 02:52:27.521970 1302 raft_consensus.cc:491] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:27.522818 1302 raft_consensus.cc:513] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.523319 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_9c3a68b2aa9a4edcb80525136f7bfa96" } new_table_name: "default.Table_f7421d0779924239a3b493954ef748b7" modify_external_catalogs: true
I20250112 02:52:27.523829 1302 leader_election.cc:290] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:27.524255 991 leader_election.cc:343] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: 3567820f0d4349799b681fdfff0ebad8
W20250112 02:52:27.527271 989 leader_election.cc:343] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 3567820f0d4349799b681fdfff0ebad8
I20250112 02:52:27.527391 989 leader_election.cc:304] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:27.528474 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 04f135b226774285a077e5806131d91b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.528956 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 04f135b226774285a077e5806131d91b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:27.529672 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_c7107062f9574066bf9a4a362f9b3cfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.530161 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7107062f9574066bf9a4a362f9b3cfb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.531023 1302 raft_consensus.cc:2743] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:27.531953 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_a0ebd60cbffb4360a1890752da449c1f" } new_table_name: "default.Table_7d3217ab87b546e88e66003083b86d4b" modify_external_catalogs: true
I20250112 02:52:27.532820 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 04f135b226774285a077e5806131d91b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.534358 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_2a2995d7964346c08fcbff96b4cc3ca5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.534970 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a2995d7964346c08fcbff96b4cc3ca5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.535145 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_34c47be51dd0466ca34b2fdc2bba8919" } new_table_name: "default.Table_ec58001618964e3bacdaeaed93191082" modify_external_catalogs: true
I20250112 02:52:27.536463 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_45c4d5f155af43818f8fe32452dcf5d8" } new_table_name: "default.Table_5fb592e41cc4406aa27546e7b41fdb88" modify_external_catalogs: true
I20250112 02:52:27.537673 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_b352d1533c5743bf8def827a46e635aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.538081 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b352d1533c5743bf8def827a46e635aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.546003 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_0f1762619e9543018fd2df57adb6367e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.546427 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f1762619e9543018fd2df57adb6367e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:27.552983 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bb4ef0220dde43569452379639759db5 has no consensus state
I20250112 02:52:27.554121 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_08d7f9f60a9043ab9712f1c1234a7969"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.554509 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08d7f9f60a9043ab9712f1c1234a7969 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:27.558965 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 1b184806b4884cd3b0a002905a1e57af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b184806b4884cd3b0a002905a1e57af already in progress: creating tablet
W20250112 02:52:27.561347 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f31b777e7e0f43a28839d8c48c6a6590 consensus state has no leader
I20250112 02:52:27.567368 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet c4a3486ef011492c8a1df8d6278adedf
W20250112 02:52:27.568308 714 consensus_peers.cc:487] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:27.568518 886 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a3486ef011492c8a1df8d6278adedf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.569496 751 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a3486ef011492c8a1df8d6278adedf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:27.569795 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet e5c10fad790046a88fbc8ebe0303c481 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5c10fad790046a88fbc8ebe0303c481 already in progress: creating tablet
W20250112 02:52:27.570035 513 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 7a34d03a80ae402db6607261a5c40b60 because the tablet was not found. No further retry: Not found: Tablet not found: 7a34d03a80ae402db6607261a5c40b60
W20250112 02:52:27.570273 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 04f135b226774285a077e5806131d91b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04f135b226774285a077e5806131d91b already in progress: creating tablet
W20250112 02:52:27.572192 513 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet c4a3486ef011492c8a1df8d6278adedf because the tablet was not found. No further retry: Not found: Tablet not found: c4a3486ef011492c8a1df8d6278adedf
I20250112 02:52:27.573311 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_2db5d877868f4917acf8889baffd14e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.573892 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2db5d877868f4917acf8889baffd14e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.576179 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_176ee2c83d9b497f9f14d05dae1131be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.576785 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_176ee2c83d9b497f9f14d05dae1131be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.577713 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8e1b96accd459eaed221ded4a453b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:27.577855 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_dc563f08a790498993b01afeec904975"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.578277 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc563f08a790498993b01afeec904975 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.578671 1171 tablet_replica.cc:331] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:27.578997 1171 raft_consensus.cc:2238] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250112 02:52:27.579387 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d45fce86e02f46d4a57af718fa2a62c8 consensus state has no leader
I20250112 02:52:27.580616 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_27f1bff714d947d5b3f42be6bfbe8be2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.581161 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27f1bff714d947d5b3f42be6bfbe8be2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.579430 1171 raft_consensus.cc:2267] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:27.582466 1171 ts_tablet_manager.cc:1905] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:27.584477 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a3486ef011492c8a1df8d6278adedf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.589844 521 master_service.cc:946] ReplaceTablet: received request to replace tablet aadb2ab1d2024f889841ac91e065e064 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:27.596143 1302 raft_consensus.cc:491] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:27.596356 1302 raft_consensus.cc:513] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.597162 1302 leader_election.cc:290] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:27.598070 991 leader_election.cc:343] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: df6ccd04880442d0990c6cb0d27956bd
W20250112 02:52:27.598210 989 leader_election.cc:343] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: df6ccd04880442d0990c6cb0d27956bd
I20250112 02:52:27.598287 991 leader_election.cc:304] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:27.598486 1302 raft_consensus.cc:2743] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:27.599638 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_f22fde4d1df34a7b955de7c6eee20c3a" } new_table_name: "default.Table_c5cad53a96d640988fc5c323a0770bae" modify_external_catalogs: true
I20250112 02:52:27.615032 1172 ts_tablet_manager.cc:1918] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:27.617324 1172 log.cc:1198] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/d3e41e307df64109a41b0c1c44d3bda6
I20250112 02:52:27.618800 1172 ts_tablet_manager.cc:1939] T d3e41e307df64109a41b0c1c44d3bda6 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:27.622998 896 tablet_service.cc:1467] Processing CreateTablet for tablet b61f16f3d601436bb4b32344de28c1a7 (DEFAULT_TABLE table=default.Table_dafd4ddc8ab34adfa999f9340216f765 [id=da4d2d7725904d5bb4a6bde668df00ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.623608 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b61f16f3d601436bb4b32344de28c1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.633708 523 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 2e9533ea72f6420bb55a53cdfc9125f7 (table default.Table_7d3217ab87b546e88e66003083b86d4b [id=586b2a68665d402ea251cb9292e19da0]). Expected version 1 got 0
W20250112 02:52:27.635635 525 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b9537fcd96eb41cc8f680075ec71137b consensus state has no leader
I20250112 02:52:27.637562 757 tablet_service.cc:1467] Processing CreateTablet for tablet b61f16f3d601436bb4b32344de28c1a7 (DEFAULT_TABLE table=default.Table_dafd4ddc8ab34adfa999f9340216f765 [id=da4d2d7725904d5bb4a6bde668df00ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.638062 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b61f16f3d601436bb4b32344de28c1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.639706 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 9163878cfeb64693a9c5493e812546af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.636718 1024 tablet_service.cc:1467] Processing CreateTablet for tablet b61f16f3d601436bb4b32344de28c1a7 (DEFAULT_TABLE table=default.Table_dafd4ddc8ab34adfa999f9340216f765 [id=da4d2d7725904d5bb4a6bde668df00ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.642988 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b61f16f3d601436bb4b32344de28c1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.648154 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet d3e41e307df64109a41b0c1c44d3bda6 (table default.Table_01a0b2e48fcd4a769a3d442390622c1e [id=4d04607bb9004f3cb1871c43d0eb47ce]) successfully deleted
I20250112 02:52:27.650650 758 tablet_service.cc:1467] Processing CreateTablet for tablet 7a34d03a80ae402db6607261a5c40b60 (DEFAULT_TABLE table=default.Table_af61980bca5e4995a77da652dcfb3f0d [id=9d7284a74b4148c0a8d20eb220ef9e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.650985 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a34d03a80ae402db6607261a5c40b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.653867 895 tablet_service.cc:1467] Processing CreateTablet for tablet 7a34d03a80ae402db6607261a5c40b60 (DEFAULT_TABLE table=default.Table_af61980bca5e4995a77da652dcfb3f0d [id=9d7284a74b4148c0a8d20eb220ef9e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.654305 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a34d03a80ae402db6607261a5c40b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.661942 1462 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4a49a5756b143fcb767c3f41463f8d9. Latest consensus state: current_term: 3 leader_uuid: "f4a49a5756b143fcb767c3f41463f8d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:27.662283 1462 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:52:27.665050 1458 raft_consensus.cc:491] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:27.665266 1458 raft_consensus.cc:513] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.666900 1458 leader_election.cc:290] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:27.667639 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d06e2e8c13464db8cf801a292bdc99" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:27.668275 855 leader_election.cc:343] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.674729 1504 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250112 02:52:27.671083 757 tablet_service.cc:1467] Processing CreateTablet for tablet 795ac7eaf81d48bb926865d9ebb388b6 (DEFAULT_TABLE table=default.Table_178713b464bb4f16970851646a710574 [id=d3f31247f0e848f8b2dfa18f3437ca63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.675357 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 795ac7eaf81d48bb926865d9ebb388b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.675415 1175 ts_tablet_manager.cc:1918] T e9fc9a4f1e8944abb76dad6c2ad53d35 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:27.676905 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d06e2e8c13464db8cf801a292bdc99" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:27.677687 854 leader_election.cc:343] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.677958 854 leader_election.cc:304] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:27.678524 1458 raft_consensus.cc:2743] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:27.680637 897 tablet_service.cc:1467] Processing CreateTablet for tablet 795ac7eaf81d48bb926865d9ebb388b6 (DEFAULT_TABLE table=default.Table_178713b464bb4f16970851646a710574 [id=d3f31247f0e848f8b2dfa18f3437ca63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.681087 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 795ac7eaf81d48bb926865d9ebb388b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.689671 1179 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f4a49a5756b143fcb767c3f41463f8d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:27.692610 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_c90865422da541e8a8665d0ba0c3fdb5" } modify_external_catalogs: true
I20250112 02:52:27.692812 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_c90865422da541e8a8665d0ba0c3fdb5" } modify_external_catalogs: true
I20250112 02:52:27.690688 1179 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: This master's current role is: LEADER
I20250112 02:52:27.690460 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_dafd4ddc8ab34adfa999f9340216f765" } modify_external_catalogs: true
I20250112 02:52:27.695689 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_dafd4ddc8ab34adfa999f9340216f765" } modify_external_catalogs: true
I20250112 02:52:27.700791 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_76e4d0c54155412296bc900e03cd40bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.701331 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76e4d0c54155412296bc900e03cd40bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.702422 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_a98f3e8602ec4ba4a771b14df9c07131" } modify_external_catalogs: true
I20250112 02:52:27.702564 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_a98f3e8602ec4ba4a771b14df9c07131" } modify_external_catalogs: true
I20250112 02:52:27.708930 525 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 6a6dde41543f4c66be9f5b7fc9c4959f
I20250112 02:52:27.679286 1175 log.cc:1198] T e9fc9a4f1e8944abb76dad6c2ad53d35 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/e9fc9a4f1e8944abb76dad6c2ad53d35
I20250112 02:52:27.711009 1175 ts_tablet_manager.cc:1939] T e9fc9a4f1e8944abb76dad6c2ad53d35 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:27.724375 1384 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" instance_seqno: 1736650341878407) as {username='slave'} at 127.0.104.1:46931; Asking this server to re-register.
I20250112 02:52:27.722103 1181 tablet_bootstrap.cc:492] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:27.727381 1458 raft_consensus.cc:491] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:27.728812 1181 ts_tablet_manager.cc:1397] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.291s user 0.017s sys 0.040s
I20250112 02:52:27.728896 1458 raft_consensus.cc:513] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.729851 1458 leader_election.cc:290] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:27.729789 1181 raft_consensus.cc:357] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.730003 1181 raft_consensus.cc:383] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.730089 1181 raft_consensus.cc:738] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:27.730336 1181 consensus_queue.cc:260] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.730799 1181 ts_tablet_manager.cc:1428] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:27.731199 1181 tablet_bootstrap.cc:492] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:27.733083 1181 tablet_bootstrap.cc:654] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:27.737744 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 210) took 1993 ms. Trace:
I20250112 02:52:27.738155 1175 rpcz_store.cc:276] 0112 02:52:25.744437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:25.744493 (+ 56us) service_pool.cc:225] Handling call
0112 02:52:27.325956 (+1581463us) ts_tablet_manager.cc:1145] Deleting tablet e9fc9a4f1e8944abb76dad6c2ad53d35
0112 02:52:27.325970 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:27.411202 (+ 85232us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:27.675120 (+263918us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:27.737689 (+ 62569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1563452}
I20250112 02:52:27.738899 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet e9fc9a4f1e8944abb76dad6c2ad53d35 (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]) successfully deleted
I20250112 02:52:27.739439 1175 tablet_replica.cc:331] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:27.739780 1175 raft_consensus.cc:2238] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:27.740453 1175 pending_rounds.cc:62] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d: Trying to abort 1 pending ops.
I20250112 02:52:27.741267 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_abf2c7b9b8da40c6b9a8381b4e41bda6" } modify_external_catalogs: true
I20250112 02:52:27.741374 1175 pending_rounds.cc:69] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319820610228224 op_type: NO_OP noop_request { }
I20250112 02:52:27.741458 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_abf2c7b9b8da40c6b9a8381b4e41bda6" } modify_external_catalogs: true
I20250112 02:52:27.741532 1175 raft_consensus.cc:2883] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:27.741673 1175 raft_consensus.cc:2267] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:27.743870 897 tablet_service.cc:1467] Processing CreateTablet for tablet b9c006281dc949119951c74e23167d7f (DEFAULT_TABLE table=default.Table_1a6ba73a53b14addbacc6a2ff0711256 [id=fe95e997b21346a5b36675a9c4d25a06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.743947 519 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 2e9533ea72f6420bb55a53cdfc9125f7 (table default.Table_7d3217ab87b546e88e66003083b86d4b [id=586b2a68665d402ea251cb9292e19da0]). Expected version 1 got 0
I20250112 02:52:27.744454 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9c006281dc949119951c74e23167d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.749548 893 tablet_service.cc:1467] Processing CreateTablet for tablet 763f77385e404c149c16edabe469c299 (DEFAULT_TABLE table=default.Table_6ff2131cc2144af7b6c223f14e9eb25a [id=fc8a29e5f2e24a4784009e85a98a40bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.749881 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 763f77385e404c149c16edabe469c299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.750276 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet b61f16f3d601436bb4b32344de28c1a7
I20250112 02:52:27.750833 1175 ts_tablet_manager.cc:1905] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:27.751904 825 heartbeater.cc:463] Registering TS with master...
I20250112 02:52:27.752102 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba1744138fe42ae9d097db01cac083d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:27.752628 855 leader_election.cc:343] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.753645 886 tablet_service.cc:1514] Processing DeleteTablet for tablet b61f16f3d601436bb4b32344de28c1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.754684 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_b352d1533c5743bf8def827a46e635aa" } new_table_name: "default.Table_bc3feb9849cf4e5db05a15330115dd52" modify_external_catalogs: true
I20250112 02:52:27.755342 1384 ts_manager.cc:194] Registered new tserver with Master: 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:27.757879 1181 tablet_bootstrap.cc:492] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:27.758031 1181 ts_tablet_manager.cc:1397] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.005s
I20250112 02:52:27.758823 1181 raft_consensus.cc:357] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:27.759028 1181 raft_consensus.cc:383] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.761132 754 tablet_service.cc:1514] Processing DeleteTablet for tablet b61f16f3d601436bb4b32344de28c1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:27.761168 1181 raft_consensus.cc:738] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:27.761415 1181 consensus_queue.cc:260] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:27.761927 1181 ts_tablet_manager.cc:1428] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250112 02:52:27.762279 1181 tablet_bootstrap.cc:492] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:27.764139 1181 tablet_bootstrap.cc:654] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:27.765421 521 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ab2e31a65294407bb62b13e92898a7d2 (table default.Table_f7d4187a2ec34710a2e1c52c00be1fb8 [id=f8cd4289f89a4a748c26a7a01236406e]). Expected version 1 got 0
I20250112 02:52:27.766548 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_176ee2c83d9b497f9f14d05dae1131be" } new_table_name: "default.Table_74701c1528924a7b840a89c1dd7eda96" modify_external_catalogs: true
I20250112 02:52:27.767561 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_08d7f9f60a9043ab9712f1c1234a7969" } new_table_name: "default.Table_31424b670f274fd3abd17eb8451ba95c" modify_external_catalogs: true
I20250112 02:52:27.767894 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_e14b6c6d907942ebb7f8410e18d38ce2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.768419 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e14b6c6d907942ebb7f8410e18d38ce2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.771248 1175 ts_tablet_manager.cc:1918] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:27.771450 526 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_1a6ba73a53b14addbacc6a2ff0711256" } modify_external_catalogs: true
I20250112 02:52:27.772250 526 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_1a6ba73a53b14addbacc6a2ff0711256" } modify_external_catalogs: true
I20250112 02:52:27.774771 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_5adbd32e9a0c4736844af4c72b7787fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.775264 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5adbd32e9a0c4736844af4c72b7787fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.775807 757 tablet_service.cc:1467] Processing CreateTablet for tablet 763f77385e404c149c16edabe469c299 (DEFAULT_TABLE table=default.Table_6ff2131cc2144af7b6c223f14e9eb25a [id=fc8a29e5f2e24a4784009e85a98a40bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.776190 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 763f77385e404c149c16edabe469c299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.779711 758 tablet_service.cc:1467] Processing CreateTablet for tablet b9c006281dc949119951c74e23167d7f (DEFAULT_TABLE table=default.Table_1a6ba73a53b14addbacc6a2ff0711256 [id=fe95e997b21346a5b36675a9c4d25a06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.780095 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9c006281dc949119951c74e23167d7f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:27.780380 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet c7c530aa0aa64e47ae27d61f341d2bd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7c530aa0aa64e47ae27d61f341d2bd9 already in progress: creating tablet
I20250112 02:52:27.783501 1503 raft_consensus.cc:491] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:27.783725 1503 raft_consensus.cc:513] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.784669 1503 leader_election.cc:290] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:27.784916 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6dc79f4d1194f81963a8094ed6b0d2e" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:27.785148 919 raft_consensus.cc:2463] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:27.785255 1175 log.cc:1198] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/fb4c004397ee480d897e8b125d248829
I20250112 02:52:27.785704 1175 ts_tablet_manager.cc:1939] T fb4c004397ee480d897e8b125d248829 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:27.786090 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 212) took 1976 ms. Trace:
I20250112 02:52:27.786185 1175 rpcz_store.cc:276] 0112 02:52:25.810125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:25.810173 (+ 48us) service_pool.cc:225] Handling call
0112 02:52:27.739268 (+1929095us) ts_tablet_manager.cc:1145] Deleting tablet fb4c004397ee480d897e8b125d248829
0112 02:52:27.739280 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:27.768186 (+ 28906us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:27.771133 (+ 2947us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:27.786060 (+ 14927us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1923511}
I20250112 02:52:27.786513 1175 tablet_replica.cc:331] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:27.786744 713 leader_election.cc:304] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:27.786784 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet fb4c004397ee480d897e8b125d248829 (table default.Table_091c87de29164fc784158659eeaf545a [id=b8c138bea585437c83ee0ed94c008284]) successfully deleted
I20250112 02:52:27.787335 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 7a34d03a80ae402db6607261a5c40b60 (DEFAULT_TABLE table=default.Table_af61980bca5e4995a77da652dcfb3f0d [id=9d7284a74b4148c0a8d20eb220ef9e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.788780 1503 raft_consensus.cc:2798] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:27.789083 1503 raft_consensus.cc:491] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:27.789369 1503 raft_consensus.cc:3054] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:27.788473 1181 tablet_bootstrap.cc:492] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:27.791671 1181 ts_tablet_manager.cc:1397] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
W20250112 02:52:27.791321 855 consensus_peers.cc:487] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:27.792536 1181 raft_consensus.cc:357] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.792732 1181 raft_consensus.cc:383] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.792812 1181 raft_consensus.cc:738] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:27.793046 1181 consensus_queue.cc:260] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.793114 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba1744138fe42ae9d097db01cac083d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:27.793599 1181 ts_tablet_manager.cc:1428] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:27.793893 1181 tablet_bootstrap.cc:492] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
W20250112 02:52:27.793928 854 leader_election.cc:343] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.794061 854 leader_election.cc:304] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:27.794494 1457 raft_consensus.cc:2743] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:27.795790 1181 tablet_bootstrap.cc:654] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:27.797446 1175 raft_consensus.cc:2238] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:27.797909 1175 raft_consensus.cc:2267] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:27.799579 1175 ts_tablet_manager.cc:1905] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:27.805532 1384 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" instance_seqno: 1736650342312772) as {username='slave'} at 127.0.104.2:55403; Asking this server to re-register.
I20250112 02:52:27.805730 1177 tablet_bootstrap.cc:492] T a47cdc208fbc431b8326364e84432cf5 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:27.805863 1177 ts_tablet_manager.cc:1397] T a47cdc208fbc431b8326364e84432cf5 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.479s user 0.110s sys 0.199s
I20250112 02:52:27.806661 1171 ts_tablet_manager.cc:1918] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:27.808995 1175 ts_tablet_manager.cc:1918] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:27.809137 1175 log.cc:1198] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/9427951452104a7db765437493b21e90
I20250112 02:52:27.814131 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_03ccb002507e4b93b1b413ae70602cfc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.814702 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03ccb002507e4b93b1b413ae70602cfc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.815014 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 795ac7eaf81d48bb926865d9ebb388b6 (DEFAULT_TABLE table=default.Table_178713b464bb4f16970851646a710574 [id=d3f31247f0e848f8b2dfa18f3437ca63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.818215 524 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 2e9533ea72f6420bb55a53cdfc9125f7 (table default.Table_7d3217ab87b546e88e66003083b86d4b [id=586b2a68665d402ea251cb9292e19da0]). Expected version 1 got 0
I20250112 02:52:27.818467 1177 raft_consensus.cc:357] T a47cdc208fbc431b8326364e84432cf5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.818655 1177 raft_consensus.cc:383] T a47cdc208fbc431b8326364e84432cf5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.818745 1177 raft_consensus.cc:738] T a47cdc208fbc431b8326364e84432cf5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:27.818939 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_ec85ae9d8f3c49699f712c3104096f2b" } new_table_name: "default.Table_e404faebac7b4fa79ca69a843876490d" modify_external_catalogs: true
I20250112 02:52:27.818958 1177 consensus_queue.cc:260] T a47cdc208fbc431b8326364e84432cf5 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.819572 1177 ts_tablet_manager.cc:1428] T a47cdc208fbc431b8326364e84432cf5 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250112 02:52:27.819782 1171 log.cc:1198] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/5a8e1b96accd459eaed221ded4a453b6
I20250112 02:52:27.821343 1177 tablet_bootstrap.cc:492] T 97b31ebe035c4552baaffc3133ec73c1 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:27.821300 1171 ts_tablet_manager.cc:1939] T 5a8e1b96accd459eaed221ded4a453b6 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:27.822206 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 2730634a5cfd4c2fb5c4e30100c556d4
I20250112 02:52:27.822891 1177 tablet_bootstrap.cc:654] T 97b31ebe035c4552baaffc3133ec73c1 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:27.823742 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 87759e8b99b04e628badc8f60cb38bb3
W20250112 02:52:27.825452 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 has no consensus state
I20250112 02:52:27.825917 1175 ts_tablet_manager.cc:1939] T 9427951452104a7db765437493b21e90 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:27.826167 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_9f651533b2a6469aa03a744720bdc14a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.826619 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f651533b2a6469aa03a744720bdc14a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.826675 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 219) took 1650 ms. Trace:
I20250112 02:52:27.826752 1175 rpcz_store.cc:276] 0112 02:52:26.176343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.176405 (+ 62us) service_pool.cc:225] Handling call
0112 02:52:27.786445 (+1610040us) ts_tablet_manager.cc:1145] Deleting tablet 9427951452104a7db765437493b21e90
0112 02:52:27.786457 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:27.805839 (+ 19382us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:27.808903 (+ 3064us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:27.826626 (+ 17723us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1609262}
I20250112 02:52:27.827805 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_e495ee84abe3479c80c8ab2e14f86c96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:27.810119 964 heartbeater.cc:463] Registering TS with master...
I20250112 02:52:27.818213 1181 tablet_bootstrap.cc:492] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:27.828176 1181 ts_tablet_manager.cc:1397] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
W20250112 02:52:27.828261 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e495ee84abe3479c80c8ab2e14f86c96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.828912 1181 raft_consensus.cc:357] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.829104 1181 raft_consensus.cc:383] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.829205 1181 raft_consensus.cc:738] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:27.829445 1181 consensus_queue.cc:260] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.829774 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_d86beb0720a94db0ab6452f1b197cdc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:27.830014 1181 ts_tablet_manager.cc:1428] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250112 02:52:27.830181 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d86beb0720a94db0ab6452f1b197cdc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.830266 1181 tablet_bootstrap.cc:492] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:27.832412 1181 tablet_bootstrap.cc:654] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:27.833609 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 5a8e1b96accd459eaed221ded4a453b6 (table default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94]) successfully deleted
W20250112 02:52:27.833886 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b61f16f3d601436bb4b32344de28c1a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b61f16f3d601436bb4b32344de28c1a7 already in progress: creating tablet
I20250112 02:52:27.834539 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 2730634a5cfd4c2fb5c4e30100c556d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:27.834776 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 87759e8b99b04e628badc8f60cb38bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:27.834859 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 9427951452104a7db765437493b21e90 (table default.Table_d38ef91002ba4653916c476d1df3a9c6 [id=fd0dc6c5d5d14251a3e6b52e3d66ed0e]) successfully deleted
I20250112 02:52:27.835160 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 87759e8b99b04e628badc8f60cb38bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:27.835255 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 70715db0394d46d9b6b289160b00be7d has no consensus state
I20250112 02:52:27.835373 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 2730634a5cfd4c2fb5c4e30100c556d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.836283 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 224) took 1567 ms. Trace:
I20250112 02:52:27.836390 1175 rpcz_store.cc:276] 0112 02:52:26.269234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.269449 (+ 215us) service_pool.cc:225] Handling call
0112 02:52:27.836093 (+1566644us) ts_tablet_manager.cc:1145] Deleting tablet 5dd48d0e68014e1aa0c44620527baec9
0112 02:52:27.836101 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:27.836223 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1564144}
W20250112 02:52:27.836818 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 5dd48d0e68014e1aa0c44620527baec9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5dd48d0e68014e1aa0c44620527baec9 already in progress: creating tablet
I20250112 02:52:27.837338 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_254eb676d963466fb32f0df6af436598"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.837774 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_254eb676d963466fb32f0df6af436598 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.839051 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 223) took 1569 ms. Trace:
I20250112 02:52:27.839145 1175 rpcz_store.cc:276] 0112 02:52:26.269149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.269429 (+ 280us) service_pool.cc:225] Handling call
0112 02:52:27.838896 (+1569467us) ts_tablet_manager.cc:1145] Deleting tablet 6913b0ac9ceb4dbc9cdf591f159134f7
0112 02:52:27.838904 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:27.838998 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1563286}
W20250112 02:52:27.839323 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 2730634a5cfd4c2fb5c4e30100c556d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2730634a5cfd4c2fb5c4e30100c556d4 already in progress: creating tablet
W20250112 02:52:27.839553 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6913b0ac9ceb4dbc9cdf591f159134f7 already in progress: creating tablet
W20250112 02:52:27.839805 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 87759e8b99b04e628badc8f60cb38bb3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87759e8b99b04e628badc8f60cb38bb3 already in progress: creating tablet
I20250112 02:52:27.840077 1020 tablet_service.cc:1467] Processing CreateTablet for tablet b9c006281dc949119951c74e23167d7f (DEFAULT_TABLE table=default.Table_1a6ba73a53b14addbacc6a2ff0711256 [id=fe95e997b21346a5b36675a9c4d25a06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.840277 757 tablet_service.cc:1514] Processing DeleteTablet for tablet a47cdc208fbc431b8326364e84432cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:27.841374 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 763f77385e404c149c16edabe469c299 (DEFAULT_TABLE table=default.Table_6ff2131cc2144af7b6c223f14e9eb25a [id=fc8a29e5f2e24a4784009e85a98a40bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.841612 1171 tablet_replica.cc:331] T a47cdc208fbc431b8326364e84432cf5 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:27.841754 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet b61f16f3d601436bb4b32344de28c1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.842257 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6dc79f4d1194f81963a8094ed6b0d2e" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:27.842904 714 leader_election.cc:343] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.842998 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 220) took 1572 ms. Trace:
I20250112 02:52:27.843083 1175 rpcz_store.cc:276] 0112 02:52:26.268580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.268797 (+ 217us) service_pool.cc:225] Handling call
0112 02:52:27.840901 (+1572104us) ts_tablet_manager.cc:1145] Deleting tablet 1c639a2bcd4740e4be7b2b162004669a
0112 02:52:27.840909 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:27.841038 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1565156}
I20250112 02:52:27.843123 1171 raft_consensus.cc:2238] T a47cdc208fbc431b8326364e84432cf5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250112 02:52:27.843497 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 1c639a2bcd4740e4be7b2b162004669a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c639a2bcd4740e4be7b2b162004669a already in progress: creating tablet
I20250112 02:52:27.843804 1171 raft_consensus.cc:2267] T a47cdc208fbc431b8326364e84432cf5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:27.845118 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 222) took 1576 ms. Trace:
I20250112 02:52:27.845216 1175 rpcz_store.cc:276] 0112 02:52:26.269038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.269405 (+ 367us) service_pool.cc:225] Handling call
0112 02:52:27.844231 (+1574826us) ts_tablet_manager.cc:1145] Deleting tablet 35f59617b34046568dd83bbb7cdc42d2
0112 02:52:27.844985 (+ 754us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:27.845063 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1566716}
W20250112 02:52:27.845574 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 35f59617b34046568dd83bbb7cdc42d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35f59617b34046568dd83bbb7cdc42d2 already in progress: creating tablet
I20250112 02:52:27.846662 1175 tablet_replica.cc:331] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:27.846869 1175 raft_consensus.cc:2238] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:27.847054 1175 raft_consensus.cc:2267] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:27.846884 1515 tablet.cc:1721] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:27.847587 1175 ts_tablet_manager.cc:1905] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:27.848438 1172 tablet_replica.cc:331] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:27.849370 1172 raft_consensus.cc:2238] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:27.853693 1181 tablet_bootstrap.cc:492] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:27.853876 1181 ts_tablet_manager.cc:1397] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.004s
I20250112 02:52:27.854682 1181 raft_consensus.cc:357] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.854899 1181 raft_consensus.cc:383] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.854995 1181 raft_consensus.cc:738] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:27.855233 1181 consensus_queue.cc:260] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.855778 1181 ts_tablet_manager.cc:1428] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:27.855311 1503 raft_consensus.cc:513] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.857178 1181 tablet_bootstrap.cc:492] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:27.860188 1384 ts_manager.cc:194] Registered new tserver with Master: fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:27.862182 1171 ts_tablet_manager.cc:1905] T a47cdc208fbc431b8326364e84432cf5 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:27.864521 1181 tablet_bootstrap.cc:654] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:27.872184 1175 ts_tablet_manager.cc:1918] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:27.872331 1175 log.cc:1198] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/f6803d0f553b4204b415b2734286bf4a
I20250112 02:52:27.872735 1175 ts_tablet_manager.cc:1939] T f6803d0f553b4204b415b2734286bf4a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:27.873438 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 225) took 1604 ms. Trace:
I20250112 02:52:27.873580 1175 rpcz_store.cc:276] 0112 02:52:26.269350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.269472 (+ 122us) service_pool.cc:225] Handling call
0112 02:52:27.846585 (+1577113us) ts_tablet_manager.cc:1145] Deleting tablet f6803d0f553b4204b415b2734286bf4a
0112 02:52:27.846594 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:27.862221 (+ 15627us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:27.872063 (+ 9842us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:27.873405 (+ 1342us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1568759}
I20250112 02:52:27.853693 1516 tablet.cc:1721] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:27.874008 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet f6803d0f553b4204b415b2734286bf4a (table default.Table_c9cd1883fedc45e89202fd92fbbd0e50 [id=944bc8032e5b4d32a6d568676f53a401]) successfully deleted
I20250112 02:52:27.876917 1177 tablet_bootstrap.cc:492] T 97b31ebe035c4552baaffc3133ec73c1 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:27.877665 1177 ts_tablet_manager.cc:1397] T 97b31ebe035c4552baaffc3133ec73c1 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.056s user 0.021s sys 0.031s
I20250112 02:52:27.881575 1181 tablet_bootstrap.cc:492] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:27.881701 1181 ts_tablet_manager.cc:1397] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.003s
I20250112 02:52:27.882375 1181 raft_consensus.cc:357] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.882558 1181 raft_consensus.cc:383] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.882643 1181 raft_consensus.cc:738] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:27.882834 1181 consensus_queue.cc:260] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.883319 1181 ts_tablet_manager.cc:1428] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:27.883635 1181 tablet_bootstrap.cc:492] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:27.883965 1177 raft_consensus.cc:357] T 97b31ebe035c4552baaffc3133ec73c1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.884629 1177 raft_consensus.cc:383] T 97b31ebe035c4552baaffc3133ec73c1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.884929 1177 raft_consensus.cc:738] T 97b31ebe035c4552baaffc3133ec73c1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:27.886329 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 221) took 1617 ms. Trace:
I20250112 02:52:27.886346 1181 tablet_bootstrap.cc:654] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:27.888037 1175 rpcz_store.cc:276] 0112 02:52:26.268694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.268825 (+ 131us) service_pool.cc:225] Handling call
0112 02:52:27.886187 (+1617362us) ts_tablet_manager.cc:1145] Deleting tablet 1a1e3af2883341e9b957f6ed748e604d
0112 02:52:27.886199 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:27.886298 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1607929}
W20250112 02:52:27.889479 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 1a1e3af2883341e9b957f6ed748e604d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a1e3af2883341e9b957f6ed748e604d already in progress: creating tablet
I20250112 02:52:27.889889 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 227) took 1496 ms. Trace:
I20250112 02:52:27.891062 1175 rpcz_store.cc:276] 0112 02:52:26.393348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.393405 (+ 57us) service_pool.cc:225] Handling call
0112 02:52:27.889760 (+1496355us) ts_tablet_manager.cc:1145] Deleting tablet 64707ee432fb4e96a3bbf7252dfa1813
0112 02:52:27.889769 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:27.889856 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1475666}
W20250112 02:52:27.891546 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 64707ee432fb4e96a3bbf7252dfa1813 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64707ee432fb4e96a3bbf7252dfa1813 already in progress: creating tablet
I20250112 02:52:27.891875 1175 tablet_replica.cc:331] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:27.892057 1172 raft_consensus.cc:2267] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:27.892683 1172 ts_tablet_manager.cc:1905] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:27.892971 1175 raft_consensus.cc:2238] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:27.893344 1175 raft_consensus.cc:2267] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:27.893316 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6dc79f4d1194f81963a8094ed6b0d2e" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:27.893371 512 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:39987, user_credentials={real_user=slave}} blocked reactor thread for 106139us
I20250112 02:52:27.894582 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a34d03a80ae402db6607261a5c40b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.892292 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6dc79f4d1194f81963a8094ed6b0d2e" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:27.894825 918 raft_consensus.cc:3054] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:27.897342 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 4b599e983ec04f53a440d1ab828d12ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b599e983ec04f53a440d1ab828d12ab already in progress: creating tablet
W20250112 02:52:27.897928 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b9199b0d2b924a79938a979df4196bb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9199b0d2b924a79938a979df4196bb4 already in progress: creating tablet
W20250112 02:52:27.898139 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 6cb76746b3a846c3aba531c04bf63966 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6cb76746b3a846c3aba531c04bf63966 already in progress: creating tablet
I20250112 02:52:27.898434 893 tablet_service.cc:1467] Processing CreateTablet for tablet 7320ac55dbfa47fe851fff94d32e1ffe (DEFAULT_TABLE table=default.Table_f8a40c27bebc483cbf62d5d0a4c7fb24 [id=8cc378b0da6e4b77bc2c6822cd0844fe]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:27.898350 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet a85300a9aa1c4c50a3957a004bea7933 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a85300a9aa1c4c50a3957a004bea7933 already in progress: creating tablet
I20250112 02:52:27.898756 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7320ac55dbfa47fe851fff94d32e1ffe. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:27.898808 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet ac716c6da01941a989754aac1ef430d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac716c6da01941a989754aac1ef430d5 already in progress: creating tablet
I20250112 02:52:27.898811 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 87759e8b99b04e628badc8f60cb38bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:50844
W20250112 02:52:27.899006 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 1a8a178bafcb457f9b0e9587df560493 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a8a178bafcb457f9b0e9587df560493 already in progress: creating tablet
I20250112 02:52:27.899121 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 7320ac55dbfa47fe851fff94d32e1ffe (DEFAULT_TABLE table=default.Table_f8a40c27bebc483cbf62d5d0a4c7fb24 [id=8cc378b0da6e4b77bc2c6822cd0844fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.899844 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 003a3b4705e943c6a900762ec11ddd47 (DEFAULT_TABLE table=default.Table_743b3f7029534564b35c0c137b35fd75 [id=b2fa3922f6d447769a8179052f9a1861]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.900203 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 2730634a5cfd4c2fb5c4e30100c556d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.900468 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 795ac7eaf81d48bb926865d9ebb388b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.904129 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 763f77385e404c149c16edabe469c299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.904304 886 tablet_service.cc:1467] Processing CreateTablet for tablet 003a3b4705e943c6a900762ec11ddd47 (DEFAULT_TABLE table=default.Table_743b3f7029534564b35c0c137b35fd75 [id=b2fa3922f6d447769a8179052f9a1861]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.904660 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 003a3b4705e943c6a900762ec11ddd47. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:27.905403 714 leader_election.cc:343] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.905710 1177 consensus_queue.cc:260] T 97b31ebe035c4552baaffc3133ec73c1 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.906311 1177 ts_tablet_manager.cc:1428] T 97b31ebe035c4552baaffc3133ec73c1 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20250112 02:52:27.907800 757 tablet_service.cc:1467] Processing CreateTablet for tablet 7320ac55dbfa47fe851fff94d32e1ffe (DEFAULT_TABLE table=default.Table_f8a40c27bebc483cbf62d5d0a4c7fb24 [id=8cc378b0da6e4b77bc2c6822cd0844fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.908179 758 tablet_service.cc:1467] Processing CreateTablet for tablet 003a3b4705e943c6a900762ec11ddd47 (DEFAULT_TABLE table=default.Table_743b3f7029534564b35c0c137b35fd75 [id=b2fa3922f6d447769a8179052f9a1861]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:27.914012 1503 leader_election.cc:290] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:27.915741 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6d84027ad6d40a5be5afcdd854431a5 has no consensus state
I20250112 02:52:27.917963 526 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet b9c006281dc949119951c74e23167d7f
I20250112 02:52:27.915715 1457 raft_consensus.cc:491] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:27.918514 1457 raft_consensus.cc:513] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:27.919080 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7320ac55dbfa47fe851fff94d32e1ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.919088 1457 leader_election.cc:290] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:27.919665 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb724feccc7f4fbc846fb6fa61be3385" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:27.920323 854 leader_election.cc:343] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.920368 893 tablet_service.cc:1514] Processing DeleteTablet for tablet b9c006281dc949119951c74e23167d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.915530 1175 ts_tablet_manager.cc:1905] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:27.922462 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_9dd5993a01b44a0abe33a5738c8d5c2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.922957 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9dd5993a01b44a0abe33a5738c8d5c2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.924700 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_8034217cf2a247d5af40b1123517157f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:27.925136 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8034217cf2a247d5af40b1123517157f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:27.922974 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_ab8c82be3f854aec990748d44c6b8a9d" } new_table_name: "default.Table_0b8c73e3dfb14a7fa289c2a61b2d34bc" modify_external_catalogs: true
I20250112 02:52:27.927241 918 raft_consensus.cc:2463] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:27.929852 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7320ac55dbfa47fe851fff94d32e1ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.929886 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0c2fad4f3243238641aae56b305bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:27.930362 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 003a3b4705e943c6a900762ec11ddd47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.936865 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9c006281dc949119951c74e23167d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.941910 1457 raft_consensus.cc:491] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:27.942037 1457 raft_consensus.cc:513] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:27.942621 1457 leader_election.cc:290] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:27.943248 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2e31a65294407bb62b13e92898a7d2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:27.943694 854 leader_election.cc:343] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.946398 1175 ts_tablet_manager.cc:1918] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:27.946535 1175 log.cc:1198] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/a47cdc208fbc431b8326364e84432cf5
I20250112 02:52:27.946904 1175 ts_tablet_manager.cc:1939] T a47cdc208fbc431b8326364e84432cf5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:27.947645 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 238) took 1461 ms. Trace:
I20250112 02:52:27.947743 1175 rpcz_store.cc:276] 0112 02:52:26.486641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.486706 (+ 65us) service_pool.cc:225] Handling call
0112 02:52:27.891813 (+1405107us) ts_tablet_manager.cc:1145] Deleting tablet a47cdc208fbc431b8326364e84432cf5
0112 02:52:27.891822 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:27.931696 (+ 39874us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:27.946316 (+ 14620us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:27.947595 (+ 1279us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1402407}
I20250112 02:52:27.948136 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet a47cdc208fbc431b8326364e84432cf5 (table default.Table_bd1cdf8c9b3343af8d17a9fd05c77145 [id=c19f442c5dc7459389b6bac8beda390c]) successfully deleted
I20250112 02:52:27.955410 1175 tablet_replica.cc:331] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:27.956313 1175 raft_consensus.cc:2238] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:27.957077 1175 raft_consensus.cc:2267] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:27.959738 1175 ts_tablet_manager.cc:1905] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:27.964708 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 003a3b4705e943c6a900762ec11ddd47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:27.965353 1177 tablet_bootstrap.cc:492] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:27.976909 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb724feccc7f4fbc846fb6fa61be3385" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:27.977355 1175 ts_tablet_manager.cc:1918] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:27.977453 1175 log.cc:1198] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/99715f37fbef47c6860fa1d95e350f20
I20250112 02:52:27.977804 1175 ts_tablet_manager.cc:1939] T 99715f37fbef47c6860fa1d95e350f20 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
W20250112 02:52:27.978089 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet edfd1ebe729d458ea31eb7feff7ee072 has no consensus state
I20250112 02:52:27.978489 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 239) took 1490 ms. Trace:
I20250112 02:52:27.978593 1175 rpcz_store.cc:276] 0112 02:52:26.487739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.491513 (+ 3774us) service_pool.cc:225] Handling call
0112 02:52:27.955339 (+1463826us) ts_tablet_manager.cc:1145] Deleting tablet 99715f37fbef47c6860fa1d95e350f20
0112 02:52:27.955347 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:27.969516 (+ 14169us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:27.977256 (+ 7740us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:27.978456 (+ 1200us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1463181}
I20250112 02:52:27.979020 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 99715f37fbef47c6860fa1d95e350f20 (table default.Table_b56246b266524f2eb968fe7df99ce8b2 [id=d3a0fac8c9234ccc92b4c929c973693d]) successfully deleted
I20250112 02:52:27.979712 517 master_service.cc:946] ReplaceTablet: received request to replace tablet aba1744138fe42ae9d097db01cac083d from {username='slave'} at 127.0.0.1:40934
W20250112 02:52:27.980307 855 leader_election.cc:343] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.980446 855 leader_election.cc:304] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:27.980821 1457 raft_consensus.cc:2743] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:27.981284 525 master_service.cc:946] ReplaceTablet: received request to replace tablet 416fb6820cd54e118950c68d381f51b2 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:27.982631 1181 tablet_bootstrap.cc:492] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:27.982776 1181 ts_tablet_manager.cc:1397] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.099s user 0.003s sys 0.004s
I20250112 02:52:27.983892 1181 raft_consensus.cc:357] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:27.984104 1181 raft_consensus.cc:383] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:27.984174 1181 raft_consensus.cc:738] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:27.984328 713 leader_election.cc:304] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:27.984359 1181 consensus_queue.cc:260] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:27.985347 1181 ts_tablet_manager.cc:1428] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250112 02:52:27.985323 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_f7bab317410d408aa363857a99be6e3c" } modify_external_catalogs: true
I20250112 02:52:27.985481 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_f7bab317410d408aa363857a99be6e3c" } modify_external_catalogs: true
I20250112 02:52:27.985634 1181 tablet_bootstrap.cc:492] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:27.986079 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_f4d612cd6a524bc493147493c5a4a284" } modify_external_catalogs: true
I20250112 02:52:27.986254 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_f4d612cd6a524bc493147493c5a4a284" } modify_external_catalogs: true
I20250112 02:52:27.987927 1181 tablet_bootstrap.cc:654] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:27.991108 754 tablet_service.cc:1514] Processing DeleteTablet for tablet b9c006281dc949119951c74e23167d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:27.991511 1175 tablet_replica.cc:331] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:27.991609 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2e31a65294407bb62b13e92898a7d2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:27.991698 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd498ccce484a89855c982762086890 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:27.992058 855 leader_election.cc:343] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:27.992187 855 leader_election.cc:304] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:27.992527 1457 raft_consensus.cc:2743] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:27.992597 1177 tablet_bootstrap.cc:654] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:27.991721 1175 raft_consensus.cc:2238] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:27.992841 1175 raft_consensus.cc:2267] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:27.995429 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet b9c006281dc949119951c74e23167d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:27.997269 1173 tablet_bootstrap.cc:492] T 49650a419dfd49d5a1547770d3f2fe54 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:27.997414 1173 ts_tablet_manager.cc:1397] T 49650a419dfd49d5a1547770d3f2fe54 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.579s user 0.004s sys 0.011s
I20250112 02:52:27.998252 1175 ts_tablet_manager.cc:1905] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.002398 519 master_service.cc:946] ReplaceTablet: received request to replace tablet 7f4e4f1e451b48dc9cdd780300f6888c from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:28.002732 893 tablet_service.cc:1514] Processing DeleteTablet for tablet b7bd96fb381f4e73a0081be3fc799c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:28.003417 1503 raft_consensus.cc:2798] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:28.003857 1503 raft_consensus.cc:695] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:28.004206 1503 consensus_queue.cc:237] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.014760 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 97b31ebe035c4552baaffc3133ec73c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:28.022459 523 master_service.cc:946] ReplaceTablet: received request to replace tablet d4abf59d96414e5faa7cf75ba133832c from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:28.023312 1175 ts_tablet_manager.cc:1918] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:28.023440 1175 log.cc:1198] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/54033a747d5a49b0a1c16f138158f596
I20250112 02:52:28.025627 1172 ts_tablet_manager.cc:1918] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:28.025745 1172 log.cc:1198] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/df6ccd04880442d0990c6cb0d27956bd
I20250112 02:52:28.026129 1172 ts_tablet_manager.cc:1939] T df6ccd04880442d0990c6cb0d27956bd P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:28.032023 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_0b48267c6ffc44989b5335e25e559c0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:28.032534 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b48267c6ffc44989b5335e25e559c0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:28.033885 1175 ts_tablet_manager.cc:1939] T 54033a747d5a49b0a1c16f138158f596 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:28.034610 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 241) took 1450 ms. Trace:
I20250112 02:52:28.034704 1175 rpcz_store.cc:276] 0112 02:52:26.584322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.584375 (+ 53us) service_pool.cc:225] Handling call
0112 02:52:27.991404 (+1407029us) ts_tablet_manager.cc:1145] Deleting tablet 54033a747d5a49b0a1c16f138158f596
0112 02:52:27.991416 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:28.001231 (+ 9815us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:28.023214 (+ 21983us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:28.034550 (+ 11336us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1406829}
I20250112 02:52:28.035066 1175 tablet_replica.cc:331] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:28.035239 1175 raft_consensus.cc:2238] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.035200 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 54033a747d5a49b0a1c16f138158f596 (table default.Table_4f66980b946046a4bd2531c4ab094352 [id=c11cab752ad14d53b9b09fbaaf9b6f36]) successfully deleted
I20250112 02:52:28.035406 1175 raft_consensus.cc:2267] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:28.035881 1175 ts_tablet_manager.cc:1905] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.038722 1171 ts_tablet_manager.cc:1918] T a47cdc208fbc431b8326364e84432cf5 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:28.038888 1171 log.cc:1198] T a47cdc208fbc431b8326364e84432cf5 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/a47cdc208fbc431b8326364e84432cf5
I20250112 02:52:28.042130 520 master_service.cc:946] ReplaceTablet: received request to replace tablet f234256b343f447dadfc3d1936a824ee from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:28.046926 1177 tablet_bootstrap.cc:492] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:28.047099 1177 ts_tablet_manager.cc:1397] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.141s user 0.003s sys 0.007s
I20250112 02:52:28.047955 1172 tablet_replica.cc:331] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:28.048060 1384 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f1e775296eb746db8b318f17d26820e6" instance_seqno: 1736650342780266) as {username='slave'} at 127.0.104.3:46361; Asking this server to re-register.
I20250112 02:52:28.048151 1172 raft_consensus.cc:2238] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.048697 1100 heartbeater.cc:463] Registering TS with master...
I20250112 02:52:28.048794 1172 raft_consensus.cc:2267] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:28.049286 1172 ts_tablet_manager.cc:1905] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.049440 1171 ts_tablet_manager.cc:1939] T a47cdc208fbc431b8326364e84432cf5 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:28.049435 1177 raft_consensus.cc:357] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.049690 1177 raft_consensus.cc:383] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.048144 1173 raft_consensus.cc:357] T 49650a419dfd49d5a1547770d3f2fe54 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.050530 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet a47cdc208fbc431b8326364e84432cf5 (table default.Table_bd1cdf8c9b3343af8d17a9fd05c77145 [id=c19f442c5dc7459389b6bac8beda390c]) successfully deleted
I20250112 02:52:28.050657 1171 tablet_replica.cc:331] T 97b31ebe035c4552baaffc3133ec73c1 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
W20250112 02:52:28.050786 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b9c006281dc949119951c74e23167d7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9c006281dc949119951c74e23167d7f already in progress: creating tablet
I20250112 02:52:28.050835 1177 raft_consensus.cc:738] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.051054 1177 consensus_queue.cc:260] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.051494 1177 ts_tablet_manager.cc:1428] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250112 02:52:28.051620 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet df6ccd04880442d0990c6cb0d27956bd (table default.Table_c667abf579554ea0af764de6515a18c6 [id=a8c0e9da5acc45c3ab7f67427f59e2b6]) successfully deleted
I20250112 02:52:28.051796 1177 tablet_bootstrap.cc:492] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:28.052232 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 015412a5fa304453bd8d1be1c3cd7a46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 015412a5fa304453bd8d1be1c3cd7a46 already in progress: creating tablet
I20250112 02:52:28.053444 1177 tablet_bootstrap.cc:654] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:28.053686 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet c8a7281c0c09442ebd0cc13b8496c86e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8a7281c0c09442ebd0cc13b8496c86e already in progress: creating tablet
I20250112 02:52:28.054836 1171 raft_consensus.cc:2238] T 97b31ebe035c4552baaffc3133ec73c1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.055020 1171 raft_consensus.cc:2267] T 97b31ebe035c4552baaffc3133ec73c1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:28.055737 1171 ts_tablet_manager.cc:1905] T 97b31ebe035c4552baaffc3133ec73c1 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.057556 1384 ts_manager.cc:194] Registered new tserver with Master: f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:28.058113 1173 raft_consensus.cc:383] T 49650a419dfd49d5a1547770d3f2fe54 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.058221 1173 raft_consensus.cc:738] T 49650a419dfd49d5a1547770d3f2fe54 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.058408 1173 consensus_queue.cc:260] T 49650a419dfd49d5a1547770d3f2fe54 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.060803 1173 ts_tablet_manager.cc:1428] T 49650a419dfd49d5a1547770d3f2fe54 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.063s user 0.001s sys 0.001s
I20250112 02:52:28.061676 1171 ts_tablet_manager.cc:1918] T 97b31ebe035c4552baaffc3133ec73c1 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:28.061769 1173 tablet_bootstrap.cc:492] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:28.061805 1171 log.cc:1198] T 97b31ebe035c4552baaffc3133ec73c1 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/97b31ebe035c4552baaffc3133ec73c1
I20250112 02:52:28.062126 1171 ts_tablet_manager.cc:1939] T 97b31ebe035c4552baaffc3133ec73c1 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:28.064653 1457 raft_consensus.cc:491] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.064810 1457 raft_consensus.cc:513] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.065361 1457 leader_election.cc:290] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:28.065758 513 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet edfd1ebe729d458ea31eb7feff7ee072 has no consensus state
I20250112 02:52:28.066051 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fdeaf5fa8cc4a2b97b2178d8ecbad69" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:28.066574 855 leader_election.cc:343] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:28.066844 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f924459d1150472aafb325a11825db23 has no consensus state
I20250112 02:52:28.067027 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fdeaf5fa8cc4a2b97b2178d8ecbad69" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:28.067219 1172 ts_tablet_manager.cc:1918] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 02:52:28.067479 854 leader_election.cc:343] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.067598 854 leader_election.cc:304] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:28.068038 1457 raft_consensus.cc:2743] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:28.068684 1172 log.cc:1198] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/3567820f0d4349799b681fdfff0ebad8
I20250112 02:52:28.068787 521 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_7d3217ab87b546e88e66003083b86d4b" } modify_external_catalogs: true
I20250112 02:52:28.068966 521 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_7d3217ab87b546e88e66003083b86d4b" } modify_external_catalogs: true
I20250112 02:52:28.068571 1173 tablet_bootstrap.cc:654] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:28.070133 1172 ts_tablet_manager.cc:1939] T 3567820f0d4349799b681fdfff0ebad8 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:28.070255 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 97b31ebe035c4552baaffc3133ec73c1 (table default.Table_4e7e1a3381d2412987a7061d9030f9f9 [id=2cee6e60fe7e454ab6a9f24e2f5c47cc]) successfully deleted
I20250112 02:52:28.072580 1172 tablet_replica.cc:331] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:28.073354 1172 raft_consensus.cc:2238] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.073630 1172 raft_consensus.cc:2267] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:28.075559 1172 ts_tablet_manager.cc:1905] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.082522 1173 tablet_bootstrap.cc:492] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:28.082698 1173 ts_tablet_manager.cc:1397] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.002s
I20250112 02:52:28.083612 1173 raft_consensus.cc:357] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.083909 1173 raft_consensus.cc:383] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.083984 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 3567820f0d4349799b681fdfff0ebad8 (table default.Table_3fcca85a33054d7ab388f29bba85fe35 [id=18415100e94043e99f9884b853158ae1]) successfully deleted
I20250112 02:52:28.084033 1173 raft_consensus.cc:738] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.084378 1173 consensus_queue.cc:260] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.084934 1173 ts_tablet_manager.cc:1428] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:28.085389 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 49650a419dfd49d5a1547770d3f2fe54 with delete_type TABLET_DATA_DELETED (replaced by tablet d8eb0efc5d7f4c72bdfc7531a444d42a) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:28.085687 1181 tablet_bootstrap.cc:492] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:28.085821 1181 ts_tablet_manager.cc:1397] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.100s user 0.006s sys 0.005s
I20250112 02:52:28.087365 1173 tablet_bootstrap.cc:492] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:28.087293 1181 raft_consensus.cc:357] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.087489 1181 raft_consensus.cc:383] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.087581 1181 raft_consensus.cc:738] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.087829 1181 consensus_queue.cc:260] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.088356 1181 ts_tablet_manager.cc:1428] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 02:52:28.090939 1173 tablet_bootstrap.cc:654] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:28.093871 893 tablet_service.cc:1467] Processing CreateTablet for tablet 781fe3514f4c4f80b1ee49278502ccc1 (DEFAULT_TABLE table=default.Table_76e4d0c54155412296bc900e03cd40bd [id=aecf277dcd794bf3bf27d58c9b96dd12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.094110 1175 ts_tablet_manager.cc:1918] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:28.094252 1175 log.cc:1198] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/f9186ac46ed441ee9308490e90da0e23
I20250112 02:52:28.094681 1175 ts_tablet_manager.cc:1939] T f9186ac46ed441ee9308490e90da0e23 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:28.094739 522 catalog_manager.cc:6071] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: default.Table_7d3217ab87b546e88e66003083b86d4b [id=586b2a68665d402ea251cb9292e19da0] alter complete (version 1)
I20250112 02:52:28.094808 897 tablet_service.cc:1467] Processing CreateTablet for tablet 4fc8658b0efe43bdb339edbff64973f6 (DEFAULT_TABLE table=default.Table_dc563f08a790498993b01afeec904975 [id=955093b0cd9a49bdbdc0517dc693a41d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.094908 1181 tablet_bootstrap.cc:492] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:28.095614 886 tablet_service.cc:1467] Processing CreateTablet for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (DEFAULT_TABLE table=default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.096199 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_e3301643d2294e8ea5f9c3c1bc366eb9" } new_table_name: "default.Table_009a6976f6db4f218e3c3373368a58d7" modify_external_catalogs: true
I20250112 02:52:28.096514 1181 tablet_bootstrap.cc:654] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:28.099004 1172 ts_tablet_manager.cc:1918] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:28.099089 1172 log.cc:1198] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/5a8e1b96accd459eaed221ded4a453b6
I20250112 02:52:28.099504 1172 ts_tablet_manager.cc:1939] T 5a8e1b96accd459eaed221ded4a453b6 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:28.101529 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_e3c5ba009c9f4301a3f99d51a8dcf687" } modify_external_catalogs: true
I20250112 02:52:28.101740 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_e3c5ba009c9f4301a3f99d51a8dcf687" } modify_external_catalogs: true
I20250112 02:52:28.105211 758 tablet_service.cc:1467] Processing CreateTablet for tablet 781fe3514f4c4f80b1ee49278502ccc1 (DEFAULT_TABLE table=default.Table_76e4d0c54155412296bc900e03cd40bd [id=aecf277dcd794bf3bf27d58c9b96dd12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.105597 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 781fe3514f4c4f80b1ee49278502ccc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.107753 757 tablet_service.cc:1467] Processing CreateTablet for tablet 4fc8658b0efe43bdb339edbff64973f6 (DEFAULT_TABLE table=default.Table_dc563f08a790498993b01afeec904975 [id=955093b0cd9a49bdbdc0517dc693a41d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.108031 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fc8658b0efe43bdb339edbff64973f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.108515 1173 tablet_bootstrap.cc:492] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:28.108636 1173 ts_tablet_manager.cc:1397] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.002s
I20250112 02:52:28.109292 1173 raft_consensus.cc:357] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.109493 1173 raft_consensus.cc:383] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.109587 1173 raft_consensus.cc:738] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.109827 1173 consensus_queue.cc:260] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.110285 1172 tablet_replica.cc:331] T 49650a419dfd49d5a1547770d3f2fe54 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:28.110460 1172 raft_consensus.cc:2238] T 49650a419dfd49d5a1547770d3f2fe54 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.110618 1172 raft_consensus.cc:2267] T 49650a419dfd49d5a1547770d3f2fe54 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:28.111662 1173 ts_tablet_manager.cc:1428] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 02:52:28.111788 754 tablet_service.cc:1467] Processing CreateTablet for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (DEFAULT_TABLE table=default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.112143 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.112537 1172 ts_tablet_manager.cc:1905] T 49650a419dfd49d5a1547770d3f2fe54 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.115403 1181 tablet_bootstrap.cc:492] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:28.115340 1173 tablet_bootstrap.cc:492] T 99715f37fbef47c6860fa1d95e350f20 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:28.115561 1181 ts_tablet_manager.cc:1397] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.003s
I20250112 02:52:28.116207 1181 raft_consensus.cc:357] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.116415 1181 raft_consensus.cc:383] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.116479 1181 raft_consensus.cc:738] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.116684 1181 consensus_queue.cc:260] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.117538 1181 ts_tablet_manager.cc:1428] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:28.121481 895 tablet_service.cc:1467] Processing CreateTablet for tablet ead501d76bb1412e9839fb36b9911e6f (DEFAULT_TABLE table=default.Table_2db5d877868f4917acf8889baffd14e2 [id=c1fbaa09536c4fa994c6949a3fd58c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.122399 896 tablet_service.cc:1467] Processing CreateTablet for tablet b6d84027ad6d40a5be5afcdd854431a5 (DEFAULT_TABLE table=default.Table_31424b670f274fd3abd17eb8451ba95c [id=ebc9aae273fb4bcd8bd904f2179f1dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.122491 1172 ts_tablet_manager.cc:1918] T 49650a419dfd49d5a1547770d3f2fe54 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:28.122587 1172 log.cc:1198] T 49650a419dfd49d5a1547770d3f2fe54 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/49650a419dfd49d5a1547770d3f2fe54
I20250112 02:52:28.122892 1172 ts_tablet_manager.cc:1939] T 49650a419dfd49d5a1547770d3f2fe54 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:28.123121 898 tablet_service.cc:1467] Processing CreateTablet for tablet bb4ef0220dde43569452379639759db5 (DEFAULT_TABLE table=default.Table_f7421d0779924239a3b493954ef748b7 [id=b42fa8fc65e344f989a98ae411e916ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.123375 1173 tablet_bootstrap.cc:654] T 99715f37fbef47c6860fa1d95e350f20 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:28.123608 888 tablet_service.cc:1467] Processing CreateTablet for tablet f924459d1150472aafb325a11825db23 (DEFAULT_TABLE table=default.Table_0b8c73e3dfb14a7fa289c2a61b2d34bc [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.124004 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 781fe3514f4c4f80b1ee49278502ccc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.125439 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fc8658b0efe43bdb339edbff64973f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.126690 1457 raft_consensus.cc:491] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.127876 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.129086 1181 tablet_bootstrap.cc:492] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:28.129060 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ead501d76bb1412e9839fb36b9911e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.129855 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d84027ad6d40a5be5afcdd854431a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.131748 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb4ef0220dde43569452379639759db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.132448 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f924459d1150472aafb325a11825db23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.132800 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 242) took 1546 ms. Trace:
I20250112 02:52:28.132912 1175 rpcz_store.cc:276] 0112 02:52:26.585940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.585990 (+ 50us) service_pool.cc:225] Handling call
0112 02:52:28.034979 (+1448989us) ts_tablet_manager.cc:1145] Deleting tablet f9186ac46ed441ee9308490e90da0e23
0112 02:52:28.034988 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:28.080935 (+ 45947us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:28.093965 (+ 13030us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:28.132758 (+ 38793us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1448794}
I20250112 02:52:28.133337 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 243) took 1536 ms. Trace:
I20250112 02:52:28.133476 1175 rpcz_store.cc:276] 0112 02:52:26.596428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.596800 (+ 372us) service_pool.cc:225] Handling call
0112 02:52:28.133205 (+1536405us) ts_tablet_manager.cc:1145] Deleting tablet f7cd0927c6a4401cbe69f55d4db9963a
0112 02:52:28.133215 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:28.133302 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1536038}
I20250112 02:52:28.133606 1458 raft_consensus.cc:491] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.133842 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 244) took 1537 ms. Trace:
I20250112 02:52:28.133747 1458 raft_consensus.cc:513] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.133960 1175 rpcz_store.cc:276] 0112 02:52:26.596550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.596887 (+ 337us) service_pool.cc:225] Handling call
0112 02:52:28.133703 (+1536816us) ts_tablet_manager.cc:1145] Deleting tablet 7a18132549374218893f35a4e4226b72
0112 02:52:28.133712 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:28.133807 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1536419}
I20250112 02:52:28.134202 756 tablet_service.cc:1467] Processing CreateTablet for tablet ead501d76bb1412e9839fb36b9911e6f (DEFAULT_TABLE table=default.Table_2db5d877868f4917acf8889baffd14e2 [id=c1fbaa09536c4fa994c6949a3fd58c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.134292 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 245) took 1537 ms. Trace:
I20250112 02:52:28.134390 1175 rpcz_store.cc:276] 0112 02:52:26.596659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.596950 (+ 291us) service_pool.cc:225] Handling call
0112 02:52:28.134169 (+1537219us) ts_tablet_manager.cc:1145] Deleting tablet 61593969d2234ce3a6089027b067d922
0112 02:52:28.134178 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:28.134251 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1536676}
I20250112 02:52:28.134459 1458 leader_election.cc:290] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:28.134591 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ead501d76bb1412e9839fb36b9911e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.134620 751 tablet_service.cc:1467] Processing CreateTablet for tablet b6d84027ad6d40a5be5afcdd854431a5 (DEFAULT_TABLE table=default.Table_31424b670f274fd3abd17eb8451ba95c [id=ebc9aae273fb4bcd8bd904f2179f1dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.134739 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 246) took 1537 ms. Trace:
I20250112 02:52:28.134824 1175 rpcz_store.cc:276] 0112 02:52:26.596753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.597015 (+ 262us) service_pool.cc:225] Handling call
0112 02:52:28.134586 (+1537571us) ts_tablet_manager.cc:1145] Deleting tablet 9d5b3df88a0c45dba879b3d506710614
0112 02:52:28.134596 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:28.134709 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1536954}
I20250112 02:52:28.134946 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d84027ad6d40a5be5afcdd854431a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.135078 1175 tablet_replica.cc:331] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:28.134999 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a88463991294bd8a335337f8b072c3d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:28.135252 1175 raft_consensus.cc:2238] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.135399 1175 raft_consensus.cc:2267] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:28.135699 855 leader_election.cc:343] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:28.136293 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet f7cd0927c6a4401cbe69f55d4db9963a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7cd0927c6a4401cbe69f55d4db9963a already in progress: creating tablet
W20250112 02:52:28.136485 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 7a18132549374218893f35a4e4226b72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a18132549374218893f35a4e4226b72 already in progress: creating tablet
W20250112 02:52:28.136651 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 61593969d2234ce3a6089027b067d922 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61593969d2234ce3a6089027b067d922 already in progress: creating tablet
W20250112 02:52:28.136829 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 9d5b3df88a0c45dba879b3d506710614 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d5b3df88a0c45dba879b3d506710614 already in progress: creating tablet
I20250112 02:52:28.137838 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a88463991294bd8a335337f8b072c3d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:28.138360 854 leader_election.cc:343] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.138463 854 leader_election.cc:304] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:28.138567 749 tablet_service.cc:1467] Processing CreateTablet for tablet bb4ef0220dde43569452379639759db5 (DEFAULT_TABLE table=default.Table_f7421d0779924239a3b493954ef748b7 [id=b42fa8fc65e344f989a98ae411e916ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.138820 1458 raft_consensus.cc:2743] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:28.138943 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb4ef0220dde43569452379639759db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.139429 1175 ts_tablet_manager.cc:1905] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.143783 753 tablet_service.cc:1467] Processing CreateTablet for tablet f924459d1150472aafb325a11825db23 (DEFAULT_TABLE table=default.Table_0b8c73e3dfb14a7fa289c2a61b2d34bc [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.144086 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f924459d1150472aafb325a11825db23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.147701 750 tablet_service.cc:1467] Processing CreateTablet for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b (DEFAULT_TABLE table=default.Table_1d107b48a09e41d6bb3401b01a21ee78 [id=68bab352836142c39c109e8e02553802]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.148041 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.149327 1173 tablet_bootstrap.cc:492] T 99715f37fbef47c6860fa1d95e350f20 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:28.149483 1173 ts_tablet_manager.cc:1397] T 99715f37fbef47c6860fa1d95e350f20 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.004s
I20250112 02:52:28.150259 1173 raft_consensus.cc:357] T 99715f37fbef47c6860fa1d95e350f20 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.150475 1173 raft_consensus.cc:383] T 99715f37fbef47c6860fa1d95e350f20 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.150547 1173 raft_consensus.cc:738] T 99715f37fbef47c6860fa1d95e350f20 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.150786 1173 consensus_queue.cc:260] T 99715f37fbef47c6860fa1d95e350f20 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.153471 1173 ts_tablet_manager.cc:1428] T 99715f37fbef47c6860fa1d95e350f20 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250112 02:52:28.155753 1173 tablet_bootstrap.cc:492] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:28.157305 1173 tablet_bootstrap.cc:654] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:28.126874 1457 raft_consensus.cc:513] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.171941 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0aba0139cf479b80b32f3dfc936535" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:28.174440 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0aba0139cf479b80b32f3dfc936535" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:28.180015 1173 tablet_bootstrap.cc:492] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:28.180150 1173 ts_tablet_manager.cc:1397] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.003s
I20250112 02:52:28.180868 1173 raft_consensus.cc:357] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.181070 1173 raft_consensus.cc:383] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.181165 1173 raft_consensus.cc:738] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.181378 1173 consensus_queue.cc:260] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:28.182097 855 leader_election.cc:343] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.183807 1181 tablet_bootstrap.cc:654] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:28.185480 854 leader_election.cc:343] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.185596 854 leader_election.cc:304] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:28.185801 1457 leader_election.cc:290] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:28.192728 1173 ts_tablet_manager.cc:1428] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250112 02:52:28.195339 1457 raft_consensus.cc:491] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.195523 1173 tablet_bootstrap.cc:492] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:28.195524 1457 raft_consensus.cc:513] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.197482 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b521522c4784d2ab4056268823f1099" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:28.198396 1458 raft_consensus.cc:2743] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:28.198556 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b521522c4784d2ab4056268823f1099" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:28.198901 1173 tablet_bootstrap.cc:654] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:28.206195 855 leader_election.cc:343] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:28.206363 854 leader_election.cc:343] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.206444 892 tablet_service.cc:1467] Processing CreateTablet for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b (DEFAULT_TABLE table=default.Table_1d107b48a09e41d6bb3401b01a21ee78 [id=68bab352836142c39c109e8e02553802]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.206584 854 leader_election.cc:304] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:28.206844 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.207505 891 tablet_service.cc:1467] Processing CreateTablet for tablet afd6d31155034908ac44a90c3a2ea7d6 (DEFAULT_TABLE table=default.Table_24cecfd421cd4047972cc0a6255ef0c0 [id=5cec72cf833347f3bfbca061f8feca41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.207921 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afd6d31155034908ac44a90c3a2ea7d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.208592 1458 raft_consensus.cc:2743] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:28.210877 1457 leader_election.cc:290] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:28.222782 1173 tablet_bootstrap.cc:492] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:28.222934 1173 ts_tablet_manager.cc:1397] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.004s
I20250112 02:52:28.223115 757 tablet_service.cc:1467] Processing CreateTablet for tablet 70715db0394d46d9b6b289160b00be7d (DEFAULT_TABLE table=default.Table_bc3feb9849cf4e5db05a15330115dd52 [id=53797be8d53c45f3b805e532889be30c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.223630 1173 raft_consensus.cc:357] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:28.223830 1173 raft_consensus.cc:383] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.223906 1173 raft_consensus.cc:738] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.223407 749 tablet_service.cc:1467] Processing CreateTablet for tablet afd6d31155034908ac44a90c3a2ea7d6 (DEFAULT_TABLE table=default.Table_24cecfd421cd4047972cc0a6255ef0c0 [id=5cec72cf833347f3bfbca061f8feca41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.224134 1173 consensus_queue.cc:260] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:28.224710 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afd6d31155034908ac44a90c3a2ea7d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.225538 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70715db0394d46d9b6b289160b00be7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.227094 1173 ts_tablet_manager.cc:1428] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:28.228364 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 781fe3514f4c4f80b1ee49278502ccc1 (DEFAULT_TABLE table=default.Table_76e4d0c54155412296bc900e03cd40bd [id=aecf277dcd794bf3bf27d58c9b96dd12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.228742 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 781fe3514f4c4f80b1ee49278502ccc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.229368 1173 tablet_bootstrap.cc:492] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:28.230491 758 tablet_service.cc:1467] Processing CreateTablet for tablet edfd1ebe729d458ea31eb7feff7ee072 (DEFAULT_TABLE table=default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.230901 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edfd1ebe729d458ea31eb7feff7ee072. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.232831 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 5a8e1b96accd459eaed221ded4a453b6 (table default.Table_9d6d135049de4ecab068bd8ab4d9fad4 [id=4abf3b98488543d2933165c2f5a11a94]) successfully deleted
W20250112 02:52:28.233052 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet e5c10fad790046a88fbc8ebe0303c481 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5c10fad790046a88fbc8ebe0303c481 already in progress: creating tablet
W20250112 02:52:28.233237 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 1b184806b4884cd3b0a002905a1e57af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b184806b4884cd3b0a002905a1e57af already in progress: creating tablet
W20250112 02:52:28.233395 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 7a34d03a80ae402db6607261a5c40b60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a34d03a80ae402db6607261a5c40b60 already in progress: creating tablet
W20250112 02:52:28.233521 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 04f135b226774285a077e5806131d91b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04f135b226774285a077e5806131d91b already in progress: creating tablet
W20250112 02:52:28.233654 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet c4a3486ef011492c8a1df8d6278adedf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4a3486ef011492c8a1df8d6278adedf already in progress: creating tablet
W20250112 02:52:28.233796 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b61f16f3d601436bb4b32344de28c1a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b61f16f3d601436bb4b32344de28c1a7 already in progress: creating tablet
W20250112 02:52:28.233963 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 87759e8b99b04e628badc8f60cb38bb3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87759e8b99b04e628badc8f60cb38bb3 already in progress: creating tablet
W20250112 02:52:28.234125 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 2730634a5cfd4c2fb5c4e30100c556d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2730634a5cfd4c2fb5c4e30100c556d4 already in progress: creating tablet
W20250112 02:52:28.234305 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b9c006281dc949119951c74e23167d7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9c006281dc949119951c74e23167d7f already in progress: creating tablet
I20250112 02:52:28.234442 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 49650a419dfd49d5a1547770d3f2fe54 (table default.Table_d561e1740cc545dc8aa4babb76b45c37 [id=de27aeff6b3a49b3956498ea5015ff33]) successfully deleted
I20250112 02:52:28.238803 526 catalog_manager.cc:5526] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:28.239264 526 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b6dc79f4d1194f81963a8094ed6b0d2e (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:28.239894 1302 raft_consensus.cc:491] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.240419 1302 raft_consensus.cc:513] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.241093 1302 leader_election.cc:290] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:28.241484 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6dc79f4d1194f81963a8094ed6b0d2e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:28.242100 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 4fc8658b0efe43bdb339edbff64973f6 (DEFAULT_TABLE table=default.Table_dc563f08a790498993b01afeec904975 [id=955093b0cd9a49bdbdc0517dc693a41d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.244333 756 tablet_service.cc:1467] Processing CreateTablet for tablet 9af55dab202444358ea1a0674d37946d (DEFAULT_TABLE table=default.Table_2750897a0a504c2ca90590075d4cf54e [id=6c6f30e60e79451397841d090d11d8b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.244566 760 tablet_service.cc:1467] Processing CreateTablet for tablet a437952889494b0e820e41fdfffe708d (DEFAULT_TABLE table=default.Table_150748c55de444749d147c320c6d8c0c [id=39ac22eea1dd45de96d7e1829ad9844d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.244638 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af55dab202444358ea1a0674d37946d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.244861 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a437952889494b0e820e41fdfffe708d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.245718 1177 tablet_bootstrap.cc:492] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:28.245875 1177 ts_tablet_manager.cc:1397] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.194s user 0.014s sys 0.027s
I20250112 02:52:28.246742 1177 raft_consensus.cc:357] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.246996 1177 raft_consensus.cc:383] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.247074 1177 raft_consensus.cc:738] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.247303 1177 consensus_queue.cc:260] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.247941 1177 ts_tablet_manager.cc:1428] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250112 02:52:28.249348 752 tablet_service.cc:1467] Processing CreateTablet for tablet 06edfca60eed4dd3a2dee7b662b0970c (DEFAULT_TABLE table=default.Table_2a2995d7964346c08fcbff96b4cc3ca5 [id=70371e156f2f401ab5ee0e4e4c7545b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.249802 755 tablet_service.cc:1467] Processing CreateTablet for tablet 3e67f0a0c065498f9a644220414415b0 (DEFAULT_TABLE table=default.Table_27f1bff714d947d5b3f42be6bfbe8be2 [id=78dff38cafdc48ddbc46b91a548c2022]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.250245 759 tablet_service.cc:1467] Processing CreateTablet for tablet 42444e99aba84a7b873bc8cdf954a6dc (DEFAULT_TABLE table=default.Table_c7107062f9574066bf9a4a362f9b3cfb [id=70e9b6c9e20f40eab186f042f21ad2ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.252265 1034 tablet_service.cc:1467] Processing CreateTablet for tablet ead501d76bb1412e9839fb36b9911e6f (DEFAULT_TABLE table=default.Table_2db5d877868f4917acf8889baffd14e2 [id=c1fbaa09536c4fa994c6949a3fd58c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.252755 1030 tablet_service.cc:1467] Processing CreateTablet for tablet bb4ef0220dde43569452379639759db5 (DEFAULT_TABLE table=default.Table_f7421d0779924239a3b493954ef748b7 [id=b42fa8fc65e344f989a98ae411e916ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.253351 1177 tablet_bootstrap.cc:492] T 54033a747d5a49b0a1c16f138158f596 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:28.253695 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e67f0a0c065498f9a644220414415b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.252908 1024 tablet_service.cc:1467] Processing CreateTablet for tablet f924459d1150472aafb325a11825db23 (DEFAULT_TABLE table=default.Table_0b8c73e3dfb14a7fa289c2a61b2d34bc [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.253989 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b (DEFAULT_TABLE table=default.Table_1d107b48a09e41d6bb3401b01a21ee78 [id=68bab352836142c39c109e8e02553802]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.254096 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fc8658b0efe43bdb339edbff64973f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.256073 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 46d1beac4410422bbf1e912d35999f3d (DEFAULT_TABLE table=default.Table_ed3c26ab0711404581dcd62adb343dde [id=3a0f5ac3329d4b60befce41c91f5ae5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.256759 1029 tablet_service.cc:1467] Processing CreateTablet for tablet edfd1ebe729d458ea31eb7feff7ee072 (DEFAULT_TABLE table=default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.256968 1033 tablet_service.cc:1467] Processing CreateTablet for tablet b6d84027ad6d40a5be5afcdd854431a5 (DEFAULT_TABLE table=default.Table_31424b670f274fd3abd17eb8451ba95c [id=ebc9aae273fb4bcd8bd904f2179f1dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.257784 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ead501d76bb1412e9839fb36b9911e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.258057 1026 tablet_service.cc:1467] Processing CreateTablet for tablet afd6d31155034908ac44a90c3a2ea7d6 (DEFAULT_TABLE table=default.Table_24cecfd421cd4047972cc0a6255ef0c0 [id=5cec72cf833347f3bfbca061f8feca41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.258237 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (DEFAULT_TABLE table=default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.258409 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 70715db0394d46d9b6b289160b00be7d (DEFAULT_TABLE table=default.Table_bc3feb9849cf4e5db05a15330115dd52 [id=53797be8d53c45f3b805e532889be30c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.258483 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 777272e7159b438ebff4db93698411d0 (DEFAULT_TABLE table=default.Table_72e8ed041bf340c383458fd7c96807ec [id=a842736799014e0886c155171bb11a13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.258571 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 47a0d939fdb94590a86389f1f4693e9a (DEFAULT_TABLE table=default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.258723 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb4ef0220dde43569452379639759db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.258939 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f924459d1150472aafb325a11825db23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.260073 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edfd1ebe729d458ea31eb7feff7ee072. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.260903 754 tablet_service.cc:1467] Processing CreateTablet for tablet 777272e7159b438ebff4db93698411d0 (DEFAULT_TABLE table=default.Table_72e8ed041bf340c383458fd7c96807ec [id=a842736799014e0886c155171bb11a13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.261118 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9c845571f44c8eb8753090d63df45f (DEFAULT_TABLE table=default.Table_7077f9a718664d8b915ee4003502306a [id=6df53b1d0c7a4fa1a18b3202741630b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.261606 1181 tablet_bootstrap.cc:492] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:28.261754 1181 ts_tablet_manager.cc:1397] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.133s user 0.006s sys 0.007s
I20250112 02:52:28.262184 1175 ts_tablet_manager.cc:1918] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:28.263247 753 tablet_service.cc:1467] Processing CreateTablet for tablet 46d1beac4410422bbf1e912d35999f3d (DEFAULT_TABLE table=default.Table_ed3c26ab0711404581dcd62adb343dde [id=3a0f5ac3329d4b60befce41c91f5ae5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.263587 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 9af55dab202444358ea1a0674d37946d (DEFAULT_TABLE table=default.Table_2750897a0a504c2ca90590075d4cf54e [id=6c6f30e60e79451397841d090d11d8b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.264295 751 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9c845571f44c8eb8753090d63df45f (DEFAULT_TABLE table=default.Table_7077f9a718664d8b915ee4003502306a [id=6df53b1d0c7a4fa1a18b3202741630b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.264942 750 tablet_service.cc:1467] Processing CreateTablet for tablet 47a0d939fdb94590a86389f1f4693e9a (DEFAULT_TABLE table=default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.265290 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06edfca60eed4dd3a2dee7b662b0970c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.255435 1177 tablet_bootstrap.cc:654] T 54033a747d5a49b0a1c16f138158f596 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:28.268854 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6dc79f4d1194f81963a8094ed6b0d2e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:28.269582 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f59617b34046568dd83bbb7cdc42d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:28.272718 899 tablet_service.cc:1467] Processing CreateTablet for tablet 3e67f0a0c065498f9a644220414415b0 (DEFAULT_TABLE table=default.Table_27f1bff714d947d5b3f42be6bfbe8be2 [id=78dff38cafdc48ddbc46b91a548c2022]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.273222 889 tablet_service.cc:1467] Processing CreateTablet for tablet 42444e99aba84a7b873bc8cdf954a6dc (DEFAULT_TABLE table=default.Table_c7107062f9574066bf9a4a362f9b3cfb [id=70e9b6c9e20f40eab186f042f21ad2ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.273610 890 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9c845571f44c8eb8753090d63df45f (DEFAULT_TABLE table=default.Table_7077f9a718664d8b915ee4003502306a [id=6df53b1d0c7a4fa1a18b3202741630b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.274014 894 tablet_service.cc:1467] Processing CreateTablet for tablet a437952889494b0e820e41fdfffe708d (DEFAULT_TABLE table=default.Table_150748c55de444749d147c320c6d8c0c [id=39ac22eea1dd45de96d7e1829ad9844d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.275076 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d84027ad6d40a5be5afcdd854431a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.275352 1019 tablet_service.cc:1467] Processing CreateTablet for tablet a437952889494b0e820e41fdfffe708d (DEFAULT_TABLE table=default.Table_150748c55de444749d147c320c6d8c0c [id=39ac22eea1dd45de96d7e1829ad9844d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.275417 1457 raft_consensus.cc:491] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.275916 1173 tablet_bootstrap.cc:654] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:28.276815 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70715db0394d46d9b6b289160b00be7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.278196 918 raft_consensus.cc:2388] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1e775296eb746db8b318f17d26820e6 in current term 1: Already voted for candidate 289afc116aa74408aa5bff74e11c50d8 in this term.
I20250112 02:52:28.279655 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af55dab202444358ea1a0674d37946d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.279495 1181 raft_consensus.cc:357] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:28.280093 1181 raft_consensus.cc:383] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.280306 893 tablet_service.cc:1467] Processing CreateTablet for tablet 46d1beac4410422bbf1e912d35999f3d (DEFAULT_TABLE table=default.Table_ed3c26ab0711404581dcd62adb343dde [id=3a0f5ac3329d4b60befce41c91f5ae5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.282435 989 leader_election.cc:304] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:28.283404 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afd6d31155034908ac44a90c3a2ea7d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.283604 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46d1beac4410422bbf1e912d35999f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.285383 1181 raft_consensus.cc:738] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.285483 1175 log.cc:1198] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/97b31ebe035c4552baaffc3133ec73c1
I20250112 02:52:28.283990 896 tablet_service.cc:1467] Processing CreateTablet for tablet 70715db0394d46d9b6b289160b00be7d (DEFAULT_TABLE table=default.Table_bc3feb9849cf4e5db05a15330115dd52 [id=53797be8d53c45f3b805e532889be30c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.288419 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46d1beac4410422bbf1e912d35999f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.288594 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b9c845571f44c8eb8753090d63df45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.288684 1302 raft_consensus.cc:491] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.288233 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 06edfca60eed4dd3a2dee7b662b0970c (DEFAULT_TABLE table=default.Table_2a2995d7964346c08fcbff96b4cc3ca5 [id=70371e156f2f401ab5ee0e4e4c7545b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.290635 1181 consensus_queue.cc:260] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:28.290939 1175 ts_tablet_manager.cc:1939] T 97b31ebe035c4552baaffc3133ec73c1 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:28.291772 897 tablet_service.cc:1467] Processing CreateTablet for tablet 9af55dab202444358ea1a0674d37946d (DEFAULT_TABLE table=default.Table_2750897a0a504c2ca90590075d4cf54e [id=6c6f30e60e79451397841d090d11d8b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.292013 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.288336 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 42444e99aba84a7b873bc8cdf954a6dc (DEFAULT_TABLE table=default.Table_c7107062f9574066bf9a4a362f9b3cfb [id=70e9b6c9e20f40eab186f042f21ad2ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.286564 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 3e67f0a0c065498f9a644220414415b0 (DEFAULT_TABLE table=default.Table_27f1bff714d947d5b3f42be6bfbe8be2 [id=78dff38cafdc48ddbc46b91a548c2022]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.292425 1302 raft_consensus.cc:513] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.292708 885 tablet_service.cc:1467] Processing CreateTablet for tablet 06edfca60eed4dd3a2dee7b662b0970c (DEFAULT_TABLE table=default.Table_2a2995d7964346c08fcbff96b4cc3ca5 [id=70371e156f2f401ab5ee0e4e4c7545b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.289309 886 tablet_service.cc:1467] Processing CreateTablet for tablet 47a0d939fdb94590a86389f1f4693e9a (DEFAULT_TABLE table=default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.294106 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 777272e7159b438ebff4db93698411d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.295326 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9533ea72f6420bb55a53cdfc9125f7" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:28.295713 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e67f0a0c065498f9a644220414415b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.298569 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47a0d939fdb94590a86389f1f4693e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.299337 887 tablet_service.cc:1467] Processing CreateTablet for tablet 777272e7159b438ebff4db93698411d0 (DEFAULT_TABLE table=default.Table_72e8ed041bf340c383458fd7c96807ec [id=a842736799014e0886c155171bb11a13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.299598 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06edfca60eed4dd3a2dee7b662b0970c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.299897 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.300420 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 777272e7159b438ebff4db93698411d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.300849 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 777272e7159b438ebff4db93698411d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.301154 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af55dab202444358ea1a0674d37946d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.302999 1457 raft_consensus.cc:513] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.307060 1302 leader_election.cc:290] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:28.309535 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06edfca60eed4dd3a2dee7b662b0970c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.309772 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42444e99aba84a7b873bc8cdf954a6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.310730 1302 raft_consensus.cc:3054] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:28.315819 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e67f0a0c065498f9a644220414415b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.319708 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42444e99aba84a7b873bc8cdf954a6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.321741 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 8c821ea3ca4946c290c17135366e0cac (DEFAULT_TABLE table=default.Table_0f1762619e9543018fd2df57adb6367e [id=5f5a61b2a00d4d29bd79a696474a787b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.324026 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47a0d939fdb94590a86389f1f4693e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.328370 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7bd96fb381f4e73a0081be3fc799c36" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:28.328963 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a437952889494b0e820e41fdfffe708d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:28.331965 855 leader_election.cc:343] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.334232 1181 ts_tablet_manager.cc:1428] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.029s user 0.000s sys 0.002s
I20250112 02:52:28.334632 1181 tablet_bootstrap.cc:492] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:28.337790 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 247) took 1679 ms. Trace:
I20250112 02:52:28.337906 1175 rpcz_store.cc:276] 0112 02:52:26.612611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.612656 (+ 45us) service_pool.cc:225] Handling call
0112 02:52:28.135027 (+1522371us) ts_tablet_manager.cc:1145] Deleting tablet 97b31ebe035c4552baaffc3133ec73c1
0112 02:52:28.135036 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:28.191934 (+ 56898us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:28.262085 (+ 70151us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:28.291704 (+ 29619us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1522153}
I20250112 02:52:28.338440 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c821ea3ca4946c290c17135366e0cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.339135 898 tablet_service.cc:1467] Processing CreateTablet for tablet edfd1ebe729d458ea31eb7feff7ee072 (DEFAULT_TABLE table=default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.336122 1457 leader_election.cc:290] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:28.344548 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b9c845571f44c8eb8753090d63df45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.354054 1302 raft_consensus.cc:2743] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:28.354168 1458 raft_consensus.cc:491] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.354336 1458 raft_consensus.cc:513] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.359181 1458 leader_election.cc:290] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:28.360507 1457 raft_consensus.cc:491] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.360682 1457 raft_consensus.cc:513] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.362111 1175 tablet_replica.cc:331] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:28.363062 1457 leader_election.cc:290] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:28.364138 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42444e99aba84a7b873bc8cdf954a6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.364130 1049 raft_consensus.cc:3054] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:28.367048 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a437952889494b0e820e41fdfffe708d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.368330 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b9c845571f44c8eb8753090d63df45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.374468 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46d1beac4410422bbf1e912d35999f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.375849 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47a0d939fdb94590a86389f1f4693e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.377031 1175 raft_consensus.cc:2238] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.381306 1175 raft_consensus.cc:2267] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:28.381989 1175 ts_tablet_manager.cc:1905] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.392246 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edfd1ebe729d458ea31eb7feff7ee072. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.395577 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70715db0394d46d9b6b289160b00be7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.425477 1181 tablet_bootstrap.cc:654] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:28.457587 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0c2fad4f3243238641aae56b305bb5" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:28.458276 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd498ccce484a89855c982762086890" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:28.459273 1503 raft_consensus.cc:491] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.459936 1503 raft_consensus.cc:513] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.460793 1503 leader_election.cc:290] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:28.462121 855 leader_election.cc:343] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:28.462343 855 leader_election.cc:343] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.462616 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99715f37fbef47c6860fa1d95e350f20" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:28.462852 1054 raft_consensus.cc:2463] T 99715f37fbef47c6860fa1d95e350f20 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:28.463864 714 leader_election.cc:304] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:28.464320 1503 raft_consensus.cc:2798] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:28.464411 1503 raft_consensus.cc:491] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:28.464727 1503 raft_consensus.cc:3054] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:28.473681 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 416fb6820cd54e118950c68d381f51b2
I20250112 02:52:28.474249 1503 raft_consensus.cc:513] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:28.474253 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 194890us
I20250112 02:52:28.475205 1537 raft_consensus.cc:491] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.475359 1537 raft_consensus.cc:513] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:28.475679 1503 leader_election.cc:290] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:28.476186 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99715f37fbef47c6860fa1d95e350f20" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:28.476348 1054 raft_consensus.cc:3054] T 99715f37fbef47c6860fa1d95e350f20 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:28.476781 991 leader_election.cc:343] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: 71b5f80ed198491780cbe17e6cd3fa15
I20250112 02:52:28.477360 1537 leader_election.cc:290] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:28.478624 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7bd96fb381f4e73a0081be3fc799c36" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:28.479192 1054 raft_consensus.cc:2463] T 99715f37fbef47c6860fa1d95e350f20 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:28.479223 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd498ccce484a89855c982762086890" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:28.479712 714 leader_election.cc:304] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:28.480063 1503 raft_consensus.cc:2798] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:28.480100 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0c2fad4f3243238641aae56b305bb5" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:28.480654 513 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 174628us
I20250112 02:52:28.480924 1503 raft_consensus.cc:695] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:28.481442 1503 consensus_queue.cc:237] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.482720 1173 tablet_bootstrap.cc:492] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:28.482851 1173 ts_tablet_manager.cc:1397] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.254s user 0.003s sys 0.010s
I20250112 02:52:28.483094 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 416fb6820cd54e118950c68d381f51b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:28.484084 1173 raft_consensus.cc:357] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.484256 1173 raft_consensus.cc:383] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.484331 1173 raft_consensus.cc:738] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.484331 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 7f4e4f1e451b48dc9cdd780300f6888c
I20250112 02:52:28.484527 1173 consensus_queue.cc:260] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:28.484748 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ea075826b16475ca474e284fc6bc545 consensus state has no leader
I20250112 02:52:28.485023 1173 ts_tablet_manager.cc:1428] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250112 02:52:28.485270 1173 tablet_bootstrap.cc:492] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:28.485466 519 catalog_manager.cc:6596] ReplaceTablet: tablet 7f4e4f1e451b48dc9cdd780300f6888c deleted and replaced by tablet 2052b4536f9745f0a785f330d706c1cf
I20250112 02:52:28.487278 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 97b31ebe035c4552baaffc3133ec73c1 (table default.Table_4e7e1a3381d2412987a7061d9030f9f9 [id=2cee6e60fe7e454ab6a9f24e2f5c47cc]) successfully deleted
I20250112 02:52:28.487717 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f4e4f1e451b48dc9cdd780300f6888c with delete_type TABLET_DATA_DELETED (replaced by tablet 2052b4536f9745f0a785f330d706c1cf) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:28.489861 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 416fb6820cd54e118950c68d381f51b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 416fb6820cd54e118950c68d381f51b2 already in progress: creating tablet
I20250112 02:52:28.489955 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f4e4f1e451b48dc9cdd780300f6888c with delete_type TABLET_DATA_DELETED (replaced by tablet 2052b4536f9745f0a785f330d706c1cf) from {username='slave'} at 127.0.0.1:50844
W20250112 02:52:28.490149 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 7f4e4f1e451b48dc9cdd780300f6888c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f4e4f1e451b48dc9cdd780300f6888c already in progress: creating tablet
W20250112 02:52:28.494863 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 7f4e4f1e451b48dc9cdd780300f6888c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f4e4f1e451b48dc9cdd780300f6888c already in progress: creating tablet
I20250112 02:52:28.497566 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_e80769df10444dd5ae4bc23a8c6cc09e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:28.498265 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e80769df10444dd5ae4bc23a8c6cc09e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:28.498992 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 416fb6820cd54e118950c68d381f51b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:28.499333 1543 tablet.cc:1721] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:28.500469 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_f7d4187a2ec34710a2e1c52c00be1fb8" } new_table_name: "default.Table_3321055e22c3410eb5e5fe8be62e004c" modify_external_catalogs: true
W20250112 02:52:28.502017 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 416fb6820cd54e118950c68d381f51b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 416fb6820cd54e118950c68d381f51b2 already in progress: creating tablet
I20250112 02:52:28.501951 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_72e8ed041bf340c383458fd7c96807ec" } new_table_name: "default.Table_f180982a92d4460d94f3f67eccd27009" modify_external_catalogs: true
I20250112 02:52:28.507894 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet a12c5d58e6c246c983e8a533906ce77d
I20250112 02:52:28.508153 757 tablet_service.cc:1467] Processing CreateTablet for tablet 8c821ea3ca4946c290c17135366e0cac (DEFAULT_TABLE table=default.Table_0f1762619e9543018fd2df57adb6367e [id=5f5a61b2a00d4d29bd79a696474a787b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.508592 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c821ea3ca4946c290c17135366e0cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.508695 759 tablet_service.cc:1514] Processing DeleteTablet for tablet a12c5d58e6c246c983e8a533906ce77d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:28.508924 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet d4abf59d96414e5faa7cf75ba133832c
W20250112 02:52:28.509320 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet a12c5d58e6c246c983e8a533906ce77d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a12c5d58e6c246c983e8a533906ce77d already in progress: creating tablet
I20250112 02:52:28.509403 523 catalog_manager.cc:6596] ReplaceTablet: tablet d4abf59d96414e5faa7cf75ba133832c deleted and replaced by tablet 5a3af0ea4e954053ae89814f3e74e6b7
I20250112 02:52:28.509815 759 tablet_service.cc:1514] Processing DeleteTablet for tablet d4abf59d96414e5faa7cf75ba133832c with delete_type TABLET_DATA_DELETED (replaced by tablet 5a3af0ea4e954053ae89814f3e74e6b7) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:28.510412 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d4abf59d96414e5faa7cf75ba133832c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4abf59d96414e5faa7cf75ba133832c already in progress: creating tablet
W20250112 02:52:28.510809 854 leader_election.cc:343] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.510210 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9533ea72f6420bb55a53cdfc9125f7" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:28.512037 854 leader_election.cc:304] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:28.513140 854 leader_election.cc:343] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.514377 854 leader_election.cc:304] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:28.515627 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet a12c5d58e6c246c983e8a533906ce77d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:28.515688 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet d4abf59d96414e5faa7cf75ba133832c with delete_type TABLET_DATA_DELETED (replaced by tablet 5a3af0ea4e954053ae89814f3e74e6b7) from {username='slave'} at 127.0.0.1:50844
W20250112 02:52:28.516427 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet a12c5d58e6c246c983e8a533906ce77d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a12c5d58e6c246c983e8a533906ce77d already in progress: creating tablet
W20250112 02:52:28.516670 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet d4abf59d96414e5faa7cf75ba133832c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4abf59d96414e5faa7cf75ba133832c already in progress: creating tablet
I20250112 02:52:28.517686 1457 raft_consensus.cc:2743] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:28.517894 1458 raft_consensus.cc:491] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.519333 1173 tablet_bootstrap.cc:654] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:28.520100 1457 raft_consensus.cc:2743] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:28.520296 1458 raft_consensus.cc:513] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.521003 1458 leader_election.cc:290] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:28.521659 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ab58e45d0d44b283019240f44b66bf" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:28.522212 855 leader_election.cc:343] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:28.522377 854 leader_election.cc:343] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.522477 854 leader_election.cc:304] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:28.523437 989 leader_election.cc:304] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:28.523542 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ab58e45d0d44b283019240f44b66bf" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:28.523828 1537 raft_consensus.cc:2743] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:28.524256 1458 raft_consensus.cc:2743] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:28.524586 989 leader_election.cc:343] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 71b5f80ed198491780cbe17e6cd3fa15
W20250112 02:52:28.524611 854 leader_election.cc:343] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.524685 989 leader_election.cc:304] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:28.524729 854 leader_election.cc:304] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:28.525012 1537 raft_consensus.cc:2743] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:28.525067 1458 raft_consensus.cc:2743] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:28.525545 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 416fb6820cd54e118950c68d381f51b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:28.525602 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_f173732a21194c7aa8f9a40eb7eb07a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:28.526271 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f173732a21194c7aa8f9a40eb7eb07a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:28.527541 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_94d3a2be9a344af082b69ecfa503d4c0" } modify_external_catalogs: true
I20250112 02:52:28.527724 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_94d3a2be9a344af082b69ecfa503d4c0" } modify_external_catalogs: true
I20250112 02:52:28.530843 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f4e4f1e451b48dc9cdd780300f6888c with delete_type TABLET_DATA_DELETED (replaced by tablet 2052b4536f9745f0a785f330d706c1cf) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:28.531553 1537 raft_consensus.cc:491] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.531729 1537 raft_consensus.cc:513] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.532419 1537 leader_election.cc:290] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:28.534585 991 leader_election.cc:343] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: f9186ac46ed441ee9308490e90da0e23
I20250112 02:52:28.537470 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet f234256b343f447dadfc3d1936a824ee
I20250112 02:52:28.537950 520 catalog_manager.cc:6596] ReplaceTablet: tablet f234256b343f447dadfc3d1936a824ee deleted and replaced by tablet 82b351c563d749df806ab5d0331a4ef2
I20250112 02:52:28.538692 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet f234256b343f447dadfc3d1936a824ee with delete_type TABLET_DATA_DELETED (replaced by tablet 82b351c563d749df806ab5d0331a4ef2) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:28.539196 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_4965c2235c4e422889169067ff632d29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:28.539747 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4965c2235c4e422889169067ff632d29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:28.539783 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f234256b343f447dadfc3d1936a824ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f234256b343f447dadfc3d1936a824ee already in progress: creating tablet
I20250112 02:52:28.540912 751 tablet_service.cc:1514] Processing DeleteTablet for tablet f234256b343f447dadfc3d1936a824ee with delete_type TABLET_DATA_DELETED (replaced by tablet 82b351c563d749df806ab5d0331a4ef2) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:28.542775 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f234256b343f447dadfc3d1936a824ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f234256b343f447dadfc3d1936a824ee already in progress: creating tablet
I20250112 02:52:28.542902 898 tablet_service.cc:1514] Processing DeleteTablet for tablet a12c5d58e6c246c983e8a533906ce77d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:28.545495 885 tablet_service.cc:1467] Processing CreateTablet for tablet 8c821ea3ca4946c290c17135366e0cac (DEFAULT_TABLE table=default.Table_0f1762619e9543018fd2df57adb6367e [id=5f5a61b2a00d4d29bd79a696474a787b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.545986 890 tablet_service.cc:1514] Processing DeleteTablet for tablet f234256b343f447dadfc3d1936a824ee with delete_type TABLET_DATA_DELETED (replaced by tablet 82b351c563d749df806ab5d0331a4ef2) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:28.546414 989 leader_election.cc:343] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: f9186ac46ed441ee9308490e90da0e23
I20250112 02:52:28.546535 989 leader_election.cc:304] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:28.546905 1537 raft_consensus.cc:2743] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:28.549755 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c821ea3ca4946c290c17135366e0cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.551409 1458 raft_consensus.cc:491] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.551576 1458 raft_consensus.cc:513] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.552659 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9163878cfeb64693a9c5493e812546af" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:28.553747 1458 leader_election.cc:290] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:28.553680 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9163878cfeb64693a9c5493e812546af" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:28.553891 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 73136be25c79491784d3ea71aa9b27f2
W20250112 02:52:28.553978 855 leader_election.cc:343] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:28.554471 854 leader_election.cc:343] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.554539 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 73136be25c79491784d3ea71aa9b27f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:28.554565 854 leader_election.cc:304] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:28.554957 1458 raft_consensus.cc:2743] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:28.555768 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 99715f37fbef47c6860fa1d95e350f20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:28.555938 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 73136be25c79491784d3ea71aa9b27f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73136be25c79491784d3ea71aa9b27f2 already in progress: creating tablet
I20250112 02:52:28.556016 1171 tablet_replica.cc:331] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:28.556185 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 73136be25c79491784d3ea71aa9b27f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:28.556308 522 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet b6dc79f4d1194f81963a8094ed6b0d2e (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:28.556955 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 2e9533ea72f6420bb55a53cdfc9125f7
I20250112 02:52:28.557592 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e9533ea72f6420bb55a53cdfc9125f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:28.557825 1172 tablet_replica.cc:331] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:28.557992 1172 raft_consensus.cc:2238] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.557987 889 tablet_service.cc:1514] Processing DeleteTablet for tablet d4abf59d96414e5faa7cf75ba133832c with delete_type TABLET_DATA_DELETED (replaced by tablet 5a3af0ea4e954053ae89814f3e74e6b7) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:28.558216 1172 raft_consensus.cc:2267] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:28.558460 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab2e31a65294407bb62b13e92898a7d2 consensus state has no leader
I20250112 02:52:28.558907 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e9533ea72f6420bb55a53cdfc9125f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:28.560171 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet f9186ac46ed441ee9308490e90da0e23 with delete_type TABLET_DATA_DELETED (replaced by tablet 04f135b226774285a077e5806131d91b) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:28.560238 522 master_service.cc:946] ReplaceTablet: received request to replace tablet f6803d0f553b4204b415b2734286bf4a from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:28.561021 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 71b5f80ed198491780cbe17e6cd3fa15 with delete_type TABLET_DATA_DELETED (replaced by tablet c7c530aa0aa64e47ae27d61f341d2bd9) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:28.561241 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 99715f37fbef47c6860fa1d95e350f20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:28.562505 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e9533ea72f6420bb55a53cdfc9125f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:28.562932 1294 raft_consensus.cc:491] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.563107 1294 raft_consensus.cc:513] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.563905 1294 leader_election.cc:290] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:28.564846 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69587c5a09d40f0baba5c2f6e8a7b41" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:28.565491 714 leader_election.cc:343] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.567546 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 73136be25c79491784d3ea71aa9b27f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:28.568241 1175 ts_tablet_manager.cc:1918] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:28.569445 1171 raft_consensus.cc:2238] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:28.569717 1177 tablet_bootstrap.cc:492] T 54033a747d5a49b0a1c16f138158f596 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:28.569866 1171 pending_rounds.cc:62] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:28.569963 1171 pending_rounds.cc:69] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319827384520704 op_type: NO_OP noop_request { }
I20250112 02:52:28.570106 1171 raft_consensus.cc:2883] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:28.570216 1171 raft_consensus.cc:2267] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:28.569868 1177 ts_tablet_manager.cc:1397] T 54033a747d5a49b0a1c16f138158f596 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.317s user 0.074s sys 0.107s
I20250112 02:52:28.571089 1177 raft_consensus.cc:357] T 54033a747d5a49b0a1c16f138158f596 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.571282 1177 raft_consensus.cc:383] T 54033a747d5a49b0a1c16f138158f596 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.571367 1177 raft_consensus.cc:738] T 54033a747d5a49b0a1c16f138158f596 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.571570 1177 consensus_queue.cc:260] T 54033a747d5a49b0a1c16f138158f596 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.572084 1171 ts_tablet_manager.cc:1905] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.573779 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_03e3448d809c4fc792d0bff94a713044"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:28.574278 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03e3448d809c4fc792d0bff94a713044 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:28.575387 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_569c0e413a014e52bcb98a9f4f644479"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:28.575848 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_569c0e413a014e52bcb98a9f4f644479 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:28.576900 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_13db6eff53f94de2ad6143fd96030225"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:28.577265 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13db6eff53f94de2ad6143fd96030225 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:28.578305 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_0b8c73e3dfb14a7fa289c2a61b2d34bc" } new_table_name: "default.Table_48ca697307d74d89aa61ef5432217ca9" modify_external_catalogs: true
I20250112 02:52:28.578994 1171 ts_tablet_manager.cc:1918] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:28.579104 1171 log.cc:1198] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/99715f37fbef47c6860fa1d95e350f20
I20250112 02:52:28.579515 1171 ts_tablet_manager.cc:1939] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:28.579559 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_e1b086de31974b33b7ac824ebeb757ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:28.580020 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1b086de31974b33b7ac824ebeb757ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:28.580299 1171 tablet_replica.cc:331] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:28.580281 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 99715f37fbef47c6860fa1d95e350f20 (table default.Table_b56246b266524f2eb968fe7df99ce8b2 [id=d3a0fac8c9234ccc92b4c929c973693d]) successfully deleted
W20250112 02:52:28.580796 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 73136be25c79491784d3ea71aa9b27f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73136be25c79491784d3ea71aa9b27f2 already in progress: creating tablet
I20250112 02:52:28.580468 1171 raft_consensus.cc:2238] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:28.581329 1171 raft_consensus.cc:2267] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:28.582067 1171 ts_tablet_manager.cc:1905] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.583173 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:28.584081 522 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c69587c5a09d40f0baba5c2f6e8a7b41 (table default.Table_1216acc1fa6b4e51a55a89ffe22bcc9a [id=b5457e4343ee43618c8309e737b69ec3]). Expected version 1 got 0
I20250112 02:52:28.584945 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_6d12502c6e714726b9d807690fd21ea4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:28.585182 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 99715f37fbef47c6860fa1d95e350f20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:28.585361 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d12502c6e714726b9d807690fd21ea4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:28.588109 1175 log.cc:1198] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/fb0aba0139cf479b80b32f3dfc936535
I20250112 02:52:28.589974 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69587c5a09d40f0baba5c2f6e8a7b41" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:28.590219 918 raft_consensus.cc:2463] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
W20250112 02:52:28.590901 713 leader_election.cc:343] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 99715f37fbef47c6860fa1d95e350f20
W20250112 02:52:28.591370 713 leader_election.cc:343] T 99715f37fbef47c6860fa1d95e350f20 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 99715f37fbef47c6860fa1d95e350f20
I20250112 02:52:28.592286 713 leader_election.cc:304] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:28.592666 1503 raft_consensus.cc:2798] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:28.592767 1503 raft_consensus.cc:491] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:28.592890 1503 raft_consensus.cc:3054] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:28.595242 1503 raft_consensus.cc:513] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.595990 1503 leader_election.cc:290] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:28.596015 1175 ts_tablet_manager.cc:1939] T fb0aba0139cf479b80b32f3dfc936535 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:28.596211 1173 tablet_bootstrap.cc:492] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:28.596360 1173 ts_tablet_manager.cc:1397] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.111s user 0.013s sys 0.023s
I20250112 02:52:28.596751 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 248) took 1921 ms. Trace:
I20250112 02:52:28.596729 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69587c5a09d40f0baba5c2f6e8a7b41" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:28.596841 1175 rpcz_store.cc:276] 0112 02:52:26.675641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.675829 (+ 188us) service_pool.cc:225] Handling call
0112 02:52:28.338230 (+1662401us) ts_tablet_manager.cc:1145] Deleting tablet fb0aba0139cf479b80b32f3dfc936535
0112 02:52:28.338242 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:28.433020 (+ 94778us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:28.519836 (+ 86816us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:28.596720 (+ 76884us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1662226}
I20250112 02:52:28.597018 1173 raft_consensus.cc:357] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.597213 1173 raft_consensus.cc:383] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 02:52:28.597194 714 leader_election.cc:343] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.597194 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet fb0aba0139cf479b80b32f3dfc936535 (table default.Table_672d7452ad6d4c99ae9f2ae77b592f78 [id=33dad7ab4106404aaf33266ed1ea5635]) successfully deleted
I20250112 02:52:28.597265 1173 raft_consensus.cc:738] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.597451 1173 consensus_queue.cc:260] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.597553 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 249) took 1921 ms. Trace:
I20250112 02:52:28.572062 1177 ts_tablet_manager.cc:1428] T 54033a747d5a49b0a1c16f138158f596 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 02:52:28.597644 1175 rpcz_store.cc:276] 0112 02:52:26.675748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.676033 (+ 285us) service_pool.cc:225] Handling call
0112 02:52:28.597430 (+1921397us) ts_tablet_manager.cc:1145] Deleting tablet 8a12b225131b464494a9f02133d37aea
0112 02:52:28.597442 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:28.597525 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1921251}
I20250112 02:52:28.597905 1173 ts_tablet_manager.cc:1428] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250112 02:52:28.598001 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 8a12b225131b464494a9f02133d37aea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a12b225131b464494a9f02133d37aea already in progress: creating tablet
I20250112 02:52:28.598304 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69587c5a09d40f0baba5c2f6e8a7b41" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:28.598465 918 raft_consensus.cc:3054] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:28.598645 1177 tablet_bootstrap.cc:492] T b885247d2cb449e0a29f6f5d81518589 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:28.599735 525 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 777272e7159b438ebff4db93698411d0 consensus state has no leader
I20250112 02:52:28.601063 1173 tablet_bootstrap.cc:492] T 328e307a4b35445f92cbfab24f2ada48 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:28.604068 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 54033a747d5a49b0a1c16f138158f596 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:28.604373 1172 ts_tablet_manager.cc:1905] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.604660 1173 tablet_bootstrap.cc:654] T 328e307a4b35445f92cbfab24f2ada48 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:28.606257 1181 tablet_bootstrap.cc:492] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:28.606423 1181 ts_tablet_manager.cc:1397] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.272s user 0.005s sys 0.006s
I20250112 02:52:28.607450 1177 tablet_bootstrap.cc:654] T b885247d2cb449e0a29f6f5d81518589 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:28.611081 1172 ts_tablet_manager.cc:1918] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:28.611240 1172 log.cc:1198] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/2e9533ea72f6420bb55a53cdfc9125f7
I20250112 02:52:28.611730 1172 ts_tablet_manager.cc:1939] T 2e9533ea72f6420bb55a53cdfc9125f7 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:28.612690 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 2e9533ea72f6420bb55a53cdfc9125f7 (table default.Table_7d3217ab87b546e88e66003083b86d4b [id=586b2a68665d402ea251cb9292e19da0]) successfully deleted
I20250112 02:52:28.613190 1172 tablet_replica.cc:331] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:28.613337 918 raft_consensus.cc:2463] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:28.613698 713 leader_election.cc:304] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:28.614053 1503 raft_consensus.cc:2798] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:28.616482 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet bb217afc08304c0ea35dee3120347ba1
I20250112 02:52:28.618479 1503 raft_consensus.cc:695] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:28.618535 885 tablet_service.cc:1514] Processing DeleteTablet for tablet bb217afc08304c0ea35dee3120347ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:28.618799 1503 consensus_queue.cc:237] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.621737 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_2b5aeec3b0bf40aea08010f65e422c2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:28.622246 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b5aeec3b0bf40aea08010f65e422c2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:28.622678 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_c5cad53a96d640988fc5c323a0770bae" } new_table_name: "default.Table_9487626cbda14256af1f17d2675b5d33" modify_external_catalogs: true
W20250112 02:52:28.639348 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f924459d1150472aafb325a11825db23 consensus state has no leader
I20250112 02:52:28.639729 751 tablet_service.cc:1514] Processing DeleteTablet for tablet bb217afc08304c0ea35dee3120347ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:28.640235 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_e6b4b27b8ab44edcb86d1e93ec94ac0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:28.640682 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b9537fcd96eb41cc8f680075ec71137b consensus state has no leader
W20250112 02:52:28.640790 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6b4b27b8ab44edcb86d1e93ec94ac0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:28.643107 1172 raft_consensus.cc:2238] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.643771 1172 raft_consensus.cc:2267] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:28.617105 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet bb217afc08304c0ea35dee3120347ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:28.628506 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet fb4c004397ee480d897e8b125d248829 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:28.647711 1173 tablet_bootstrap.cc:492] T 328e307a4b35445f92cbfab24f2ada48 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:28.647944 1173 ts_tablet_manager.cc:1397] T 328e307a4b35445f92cbfab24f2ada48 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.010s
I20250112 02:52:28.648641 1458 raft_consensus.cc:491] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.648804 1458 raft_consensus.cc:513] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.648813 1173 raft_consensus.cc:357] T 328e307a4b35445f92cbfab24f2ada48 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.648999 1173 raft_consensus.cc:383] T 328e307a4b35445f92cbfab24f2ada48 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.649081 1173 raft_consensus.cc:738] T 328e307a4b35445f92cbfab24f2ada48 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.649451 1458 leader_election.cc:290] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:28.649291 1173 consensus_queue.cc:260] T 328e307a4b35445f92cbfab24f2ada48 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.650183 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f59617b34046568dd83bbb7cdc42d2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:28.650844 855 leader_election.cc:343] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.652098 1173 ts_tablet_manager.cc:1428] T 328e307a4b35445f92cbfab24f2ada48 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:28.652184 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f59617b34046568dd83bbb7cdc42d2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:28.652482 1173 tablet_bootstrap.cc:492] T 54033a747d5a49b0a1c16f138158f596 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:28.652695 854 leader_election.cc:343] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.652815 854 leader_election.cc:304] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:28.653162 1458 raft_consensus.cc:2743] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:28.654881 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 328e307a4b35445f92cbfab24f2ada48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:28.657167 1173 tablet_bootstrap.cc:654] T 54033a747d5a49b0a1c16f138158f596 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:28.657583 1175 tablet_replica.cc:331] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:28.657846 1175 raft_consensus.cc:2238] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.658038 1175 raft_consensus.cc:2267] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:28.663870 1172 ts_tablet_manager.cc:1905] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.665421 1171 ts_tablet_manager.cc:1918] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:28.666024 1175 ts_tablet_manager.cc:1905] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.666432 1171 log.cc:1198] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/2e9533ea72f6420bb55a53cdfc9125f7
I20250112 02:52:28.667016 1171 ts_tablet_manager.cc:1939] T 2e9533ea72f6420bb55a53cdfc9125f7 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:28.670133 1181 raft_consensus.cc:357] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.670598 1181 raft_consensus.cc:383] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.671195 1172 ts_tablet_manager.cc:1918] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:28.671375 1172 log.cc:1198] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/f9186ac46ed441ee9308490e90da0e23
I20250112 02:52:28.671775 1172 ts_tablet_manager.cc:1939] T f9186ac46ed441ee9308490e90da0e23 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:28.672307 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 2e9533ea72f6420bb55a53cdfc9125f7 (table default.Table_7d3217ab87b546e88e66003083b86d4b [id=586b2a68665d402ea251cb9292e19da0]) successfully deleted
I20250112 02:52:28.670713 1181 raft_consensus.cc:738] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.672818 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet f9186ac46ed441ee9308490e90da0e23 (table default.Table_d1e5cef5fa3e4d90bada176f80bd4180 [id=f93b5bb22ec1424ea047383ed9713618]) successfully deleted
I20250112 02:52:28.673141 1172 tablet_replica.cc:331] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:28.673094 1181 consensus_queue.cc:260] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.673372 1172 raft_consensus.cc:2238] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.673554 1172 raft_consensus.cc:2267] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:28.673802 513 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 99715f37fbef47c6860fa1d95e350f20 because the tablet was not found. No further retry: Not found: Tablet not found: 99715f37fbef47c6860fa1d95e350f20
I20250112 02:52:28.674332 1172 ts_tablet_manager.cc:1905] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.679293 1181 ts_tablet_manager.cc:1428] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.073s user 0.001s sys 0.002s
I20250112 02:52:28.679766 1181 tablet_bootstrap.cc:492] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:28.680961 1172 ts_tablet_manager.cc:1918] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:28.681288 1172 log.cc:1198] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/71b5f80ed198491780cbe17e6cd3fa15
I20250112 02:52:28.681336 1181 tablet_bootstrap.cc:654] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:28.681721 1172 ts_tablet_manager.cc:1939] T 71b5f80ed198491780cbe17e6cd3fa15 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:28.682582 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_bc3feb9849cf4e5db05a15330115dd52" } modify_external_catalogs: true
I20250112 02:52:28.682771 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_bc3feb9849cf4e5db05a15330115dd52" } modify_external_catalogs: true
I20250112 02:52:28.683763 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 71b5f80ed198491780cbe17e6cd3fa15 (table default.Table_247a821819994de6adac6fabd06c3a1f [id=60486b9e81254d74b8e08a1cadde13bd]) successfully deleted
I20250112 02:52:28.689352 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_d86beb0720a94db0ab6452f1b197cdc6" } new_table_name: "default.Table_c1639450ddf94784a40b8cf28f3059c4" modify_external_catalogs: true
I20250112 02:52:28.699712 1171 tablet_replica.cc:331] T 54033a747d5a49b0a1c16f138158f596 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:28.702525 1171 raft_consensus.cc:2238] T 54033a747d5a49b0a1c16f138158f596 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.705010 918 raft_consensus.cc:1270] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Refusing update from remote peer 289afc116aa74408aa5bff74e11c50d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 02:52:28.705610 1503 consensus_queue.cc:1035] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:52:28.711617 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b1d9fae290ec4281839991f9d40b105c has no consensus state
I20250112 02:52:28.712666 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_13db6eff53f94de2ad6143fd96030225" } modify_external_catalogs: true
I20250112 02:52:28.712914 1049 raft_consensus.cc:1270] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Refusing update from remote peer 289afc116aa74408aa5bff74e11c50d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 02:52:28.712890 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_13db6eff53f94de2ad6143fd96030225" } modify_external_catalogs: true
I20250112 02:52:28.713510 1294 consensus_queue.cc:1035] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:28.718984 1458 raft_consensus.cc:491] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.719134 1458 raft_consensus.cc:513] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:28.719784 1458 leader_election.cc:290] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:28.719916 522 master_service.cc:946] ReplaceTablet: received request to replace tablet 65f22a7b1b804661af3b7e785a833739 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:28.720464 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8eb0efc5d7f4c72bdfc7531a444d42a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:28.721377 855 leader_election.cc:343] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.721802 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8eb0efc5d7f4c72bdfc7531a444d42a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:28.722324 854 leader_election.cc:343] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.722424 854 leader_election.cc:304] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:28.722700 518 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 458154c399094284864a2bf77234084d
I20250112 02:52:28.722803 1458 raft_consensus.cc:2743] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:28.725176 1171 raft_consensus.cc:2267] T 54033a747d5a49b0a1c16f138158f596 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:28.726399 525 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet d8eb0efc5d7f4c72bdfc7531a444d42a (table default.Table_d561e1740cc545dc8aa4babb76b45c37 [id=de27aeff6b3a49b3956498ea5015ff33]). Expected version 1 got 0
I20250112 02:52:28.726795 1171 ts_tablet_manager.cc:1905] T 54033a747d5a49b0a1c16f138158f596 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.726763 520 catalog_manager.cc:5526] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
W20250112 02:52:28.727049 525 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet d8eb0efc5d7f4c72bdfc7531a444d42a (table default.Table_d561e1740cc545dc8aa4babb76b45c37 [id=de27aeff6b3a49b3956498ea5015ff33], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet d8eb0efc5d7f4c72bdfc7531a444d42a consensus state has no leader
I20250112 02:52:28.727134 1181 tablet_bootstrap.cc:492] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:28.727243 520 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c69587c5a09d40f0baba5c2f6e8a7b41 (table default.Table_1216acc1fa6b4e51a55a89ffe22bcc9a [id=b5457e4343ee43618c8309e737b69ec3]). Expected version 1 got 0
I20250112 02:52:28.728030 1181 ts_tablet_manager.cc:1397] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.006s
I20250112 02:52:28.729182 1181 raft_consensus.cc:357] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.729460 1181 raft_consensus.cc:383] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.729604 1181 raft_consensus.cc:738] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.729875 1181 consensus_queue.cc:260] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.730543 1181 ts_tablet_manager.cc:1428] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:28.730840 1181 tablet_bootstrap.cc:492] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:28.733925 751 tablet_service.cc:1467] Processing CreateTablet for tablet cf54f5185a0f4c70b9057309fca14e56 (DEFAULT_TABLE table=default.Table_0b48267c6ffc44989b5335e25e559c0d [id=d8a4cc77c79449da82b721854466fc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.734050 1019 tablet_service.cc:1467] Processing CreateTablet for tablet cf54f5185a0f4c70b9057309fca14e56 (DEFAULT_TABLE table=default.Table_0b48267c6ffc44989b5335e25e559c0d [id=d8a4cc77c79449da82b721854466fc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.734532 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf54f5185a0f4c70b9057309fca14e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.736021 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf54f5185a0f4c70b9057309fca14e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.736060 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 65f22a7b1b804661af3b7e785a833739
I20250112 02:52:28.736563 522 catalog_manager.cc:6596] ReplaceTablet: tablet 65f22a7b1b804661af3b7e785a833739 deleted and replaced by tablet 3d662e0ce13f447595e5f92b94de3018
I20250112 02:52:28.737174 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f22a7b1b804661af3b7e785a833739 with delete_type TABLET_DATA_DELETED (replaced by tablet 3d662e0ce13f447595e5f92b94de3018) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:28.737291 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f22a7b1b804661af3b7e785a833739 with delete_type TABLET_DATA_DELETED (replaced by tablet 3d662e0ce13f447595e5f92b94de3018) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:28.736439 1171 ts_tablet_manager.cc:1918] T 54033a747d5a49b0a1c16f138158f596 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:28.736631 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f22a7b1b804661af3b7e785a833739 with delete_type TABLET_DATA_DELETED (replaced by tablet 3d662e0ce13f447595e5f92b94de3018) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:28.738027 1171 log.cc:1198] T 54033a747d5a49b0a1c16f138158f596 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/54033a747d5a49b0a1c16f138158f596
I20250112 02:52:28.738373 1171 ts_tablet_manager.cc:1939] T 54033a747d5a49b0a1c16f138158f596 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:28.739753 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 54033a747d5a49b0a1c16f138158f596 (table default.Table_4f66980b946046a4bd2531c4ab094352 [id=c11cab752ad14d53b9b09fbaaf9b6f36]) successfully deleted
W20250112 02:52:28.740406 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet bb217afc08304c0ea35dee3120347ba1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb217afc08304c0ea35dee3120347ba1 already in progress: creating tablet
W20250112 02:52:28.740958 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 65f22a7b1b804661af3b7e785a833739 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65f22a7b1b804661af3b7e785a833739 already in progress: creating tablet
I20250112 02:52:28.742573 1177 tablet_bootstrap.cc:492] T b885247d2cb449e0a29f6f5d81518589 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:28.742751 1177 ts_tablet_manager.cc:1397] T b885247d2cb449e0a29f6f5d81518589 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.144s user 0.032s sys 0.056s
I20250112 02:52:28.743656 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 70715db0394d46d9b6b289160b00be7d
I20250112 02:52:28.744184 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 70715db0394d46d9b6b289160b00be7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:28.744424 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 70715db0394d46d9b6b289160b00be7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:28.744962 885 tablet_service.cc:1467] Processing CreateTablet for tablet cf54f5185a0f4c70b9057309fca14e56 (DEFAULT_TABLE table=default.Table_0b48267c6ffc44989b5335e25e559c0d [id=d8a4cc77c79449da82b721854466fc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.745245 753 tablet_service.cc:1467] Processing CreateTablet for tablet bab4978edea0445fb0456e88e7fb4c44 (DEFAULT_TABLE table=default.Table_9dd5993a01b44a0abe33a5738c8d5c2b [id=6a7dcf7429b24699b036c53b3005051f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.745354 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf54f5185a0f4c70b9057309fca14e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.745326 1019 tablet_service.cc:1467] Processing CreateTablet for tablet bab4978edea0445fb0456e88e7fb4c44 (DEFAULT_TABLE table=default.Table_9dd5993a01b44a0abe33a5738c8d5c2b [id=6a7dcf7429b24699b036c53b3005051f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.745582 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bab4978edea0445fb0456e88e7fb4c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.747639 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 70715db0394d46d9b6b289160b00be7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:28.748008 1181 tablet_bootstrap.cc:654] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:28.748536 757 tablet_service.cc:1467] Processing CreateTablet for tablet c73f2fa452b84be398476b8aba6c9a08 (DEFAULT_TABLE table=default.Table_254eb676d963466fb32f0df6af436598 [id=097d902a92934882bc7c016da518642a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.748574 1035 tablet_service.cc:1467] Processing CreateTablet for tablet c73f2fa452b84be398476b8aba6c9a08 (DEFAULT_TABLE table=default.Table_254eb676d963466fb32f0df6af436598 [id=097d902a92934882bc7c016da518642a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.748857 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c73f2fa452b84be398476b8aba6c9a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.749229 896 tablet_service.cc:1467] Processing CreateTablet for tablet bab4978edea0445fb0456e88e7fb4c44 (DEFAULT_TABLE table=default.Table_9dd5993a01b44a0abe33a5738c8d5c2b [id=6a7dcf7429b24699b036c53b3005051f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.749563 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bab4978edea0445fb0456e88e7fb4c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.752117 754 tablet_service.cc:1467] Processing CreateTablet for tablet e477fc157aed4563923d85d66014293a (DEFAULT_TABLE table=default.Table_e495ee84abe3479c80c8ab2e14f86c96 [id=f5eb0a253e444d76978a22250c163a54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.752460 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e477fc157aed4563923d85d66014293a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.752454 893 tablet_service.cc:1467] Processing CreateTablet for tablet c73f2fa452b84be398476b8aba6c9a08 (DEFAULT_TABLE table=default.Table_254eb676d963466fb32f0df6af436598 [id=097d902a92934882bc7c016da518642a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.752701 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c73f2fa452b84be398476b8aba6c9a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.755734 1021 tablet_service.cc:1467] Processing CreateTablet for tablet e477fc157aed4563923d85d66014293a (DEFAULT_TABLE table=default.Table_e495ee84abe3479c80c8ab2e14f86c96 [id=f5eb0a253e444d76978a22250c163a54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.756891 889 tablet_service.cc:1467] Processing CreateTablet for tablet e477fc157aed4563923d85d66014293a (DEFAULT_TABLE table=default.Table_e495ee84abe3479c80c8ab2e14f86c96 [id=f5eb0a253e444d76978a22250c163a54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.757210 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e477fc157aed4563923d85d66014293a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.759747 1031 tablet_service.cc:1467] Processing CreateTablet for tablet fea2ef4e9a63491db850bb5393938017 (DEFAULT_TABLE table=default.Table_9f651533b2a6469aa03a744720bdc14a [id=68e2f0649cb9417e89b8c6ba0d723d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.760843 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_3b1c87a5169e49b68fdf3d5c662a27bc" } modify_external_catalogs: true
I20250112 02:52:28.761019 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_3b1c87a5169e49b68fdf3d5c662a27bc" } modify_external_catalogs: true
I20250112 02:52:28.762491 759 tablet_service.cc:1467] Processing CreateTablet for tablet e4b7c2efd7ea402e8d19038f96b9675c (DEFAULT_TABLE table=default.Table_03ccb002507e4b93b1b413ae70602cfc [id=594044c9b27a441cb1002a7b35a6a846]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.762776 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4b7c2efd7ea402e8d19038f96b9675c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.762799 886 tablet_service.cc:1467] Processing CreateTablet for tablet fea2ef4e9a63491db850bb5393938017 (DEFAULT_TABLE table=default.Table_9f651533b2a6469aa03a744720bdc14a [id=68e2f0649cb9417e89b8c6ba0d723d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.763113 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fea2ef4e9a63491db850bb5393938017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.764550 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_328664e3c01f483ca8e5ff50a497f68e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:28.765091 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_328664e3c01f483ca8e5ff50a497f68e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:28.767166 1033 tablet_service.cc:1467] Processing CreateTablet for tablet e4b7c2efd7ea402e8d19038f96b9675c (DEFAULT_TABLE table=default.Table_03ccb002507e4b93b1b413ae70602cfc [id=594044c9b27a441cb1002a7b35a6a846]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:28.768172 854 consensus_peers.cc:487] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a8e1b96accd459eaed221ded4a453b6. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:28.768464 518 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b6dc79f4d1194f81963a8094ed6b0d2e (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:28.768537 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_2db5d877868f4917acf8889baffd14e2" } new_table_name: "default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3" modify_external_catalogs: true
I20250112 02:52:28.769263 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_c1639450ddf94784a40b8cf28f3059c4" } modify_external_catalogs: true
I20250112 02:52:28.769419 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_c1639450ddf94784a40b8cf28f3059c4" } modify_external_catalogs: true
I20250112 02:52:28.772197 757 tablet_service.cc:1467] Processing CreateTablet for tablet fea2ef4e9a63491db850bb5393938017 (DEFAULT_TABLE table=default.Table_9f651533b2a6469aa03a744720bdc14a [id=68e2f0649cb9417e89b8c6ba0d723d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.772526 894 tablet_service.cc:1467] Processing CreateTablet for tablet e4b7c2efd7ea402e8d19038f96b9675c (DEFAULT_TABLE table=default.Table_03ccb002507e4b93b1b413ae70602cfc [id=594044c9b27a441cb1002a7b35a6a846]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.772629 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fea2ef4e9a63491db850bb5393938017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.772881 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4b7c2efd7ea402e8d19038f96b9675c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.743969 1177 raft_consensus.cc:357] T b885247d2cb449e0a29f6f5d81518589 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:28.773412 1177 raft_consensus.cc:383] T b885247d2cb449e0a29f6f5d81518589 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.773483 1177 raft_consensus.cc:738] T b885247d2cb449e0a29f6f5d81518589 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.773718 1177 consensus_queue.cc:260] T b885247d2cb449e0a29f6f5d81518589 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:28.774199 1177 ts_tablet_manager.cc:1428] T b885247d2cb449e0a29f6f5d81518589 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.031s user 0.001s sys 0.000s
I20250112 02:52:28.778421 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 42cb7e7a41634cf1bdb978ac9d2befcd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:28.781680 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bab4978edea0445fb0456e88e7fb4c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.782105 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4b7c2efd7ea402e8d19038f96b9675c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.782160 759 tablet_service.cc:1514] Processing DeleteTablet for tablet b885247d2cb449e0a29f6f5d81518589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:28.782781 1555 mvcc.cc:204] Tried to move back new op lower bound from 7113319828289581056 to 7113319825428361216. Current Snapshot: MvccSnapshot[applied={T|T < 7113319828289581056}]
I20250112 02:52:28.785444 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e477fc157aed4563923d85d66014293a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.785720 1177 tablet_bootstrap.cc:492] T cb724feccc7f4fbc846fb6fa61be3385 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:28.788213 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 70715db0394d46d9b6b289160b00be7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70715db0394d46d9b6b289160b00be7d already in progress: creating tablet
I20250112 02:52:28.789943 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fea2ef4e9a63491db850bb5393938017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.790058 759 tablet_service.cc:1467] Processing CreateTablet for tablet 6171b8ff2bfe41d598b0cb2535799a11 (DEFAULT_TABLE table=default.Table_e14b6c6d907942ebb7f8410e18d38ce2 [id=3097d5437937423e95b710c00473b6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.790942 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6171b8ff2bfe41d598b0cb2535799a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.795665 1177 tablet_bootstrap.cc:654] T cb724feccc7f4fbc846fb6fa61be3385 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:28.796119 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c73f2fa452b84be398476b8aba6c9a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.796676 1171 tablet_replica.cc:331] T b885247d2cb449e0a29f6f5d81518589 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:28.796835 1171 raft_consensus.cc:2238] T b885247d2cb449e0a29f6f5d81518589 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.797403 1171 raft_consensus.cc:2267] T b885247d2cb449e0a29f6f5d81518589 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:28.798345 518 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b6dc79f4d1194f81963a8094ed6b0d2e (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:28.800307 890 tablet_service.cc:1467] Processing CreateTablet for tablet 6171b8ff2bfe41d598b0cb2535799a11 (DEFAULT_TABLE table=default.Table_e14b6c6d907942ebb7f8410e18d38ce2 [id=3097d5437937423e95b710c00473b6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.800786 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6171b8ff2bfe41d598b0cb2535799a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.798415 1171 ts_tablet_manager.cc:1905] T b885247d2cb449e0a29f6f5d81518589 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.802615 754 tablet_service.cc:1467] Processing CreateTablet for tablet 2052b4536f9745f0a785f330d706c1cf (DEFAULT_TABLE table=default.Table_096703f878244b8dac2dcf187fd23fd6 [id=e1779f88b20942eaa2c640ac24946fff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.802958 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2052b4536f9745f0a785f330d706c1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.804189 1175 ts_tablet_manager.cc:1918] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:28.804379 1175 log.cc:1198] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/a1a0a13696fc41ccb28218f1275a4b5d
I20250112 02:52:28.804826 1175 ts_tablet_manager.cc:1939] T a1a0a13696fc41ccb28218f1275a4b5d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:28.805574 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 259) took 1984 ms. Trace:
I20250112 02:52:28.806622 898 tablet_service.cc:1467] Processing CreateTablet for tablet 2052b4536f9745f0a785f330d706c1cf (DEFAULT_TABLE table=default.Table_096703f878244b8dac2dcf187fd23fd6 [id=e1779f88b20942eaa2c640ac24946fff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.810889 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2052b4536f9745f0a785f330d706c1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.811506 1175 rpcz_store.cc:276] 0112 02:52:26.820747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.820785 (+ 38us) service_pool.cc:225] Handling call
0112 02:52:28.598210 (+1777425us) ts_tablet_manager.cc:1145] Deleting tablet a1a0a13696fc41ccb28218f1275a4b5d
0112 02:52:28.598218 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:28.744698 (+146480us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:28.804102 (+ 59404us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:28.805514 (+ 1412us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1777234}
I20250112 02:52:28.812323 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet a1a0a13696fc41ccb28218f1275a4b5d (table default.Table_62be2ee0d4744f198098621bf776cf01 [id=bf2df21352c64c069c2a06c878911016]) successfully deleted
I20250112 02:52:28.812760 1175 tablet_replica.cc:331] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:28.813280 1175 raft_consensus.cc:2238] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.813446 1171 ts_tablet_manager.cc:1918] T b885247d2cb449e0a29f6f5d81518589 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:28.813568 1171 log.cc:1198] T b885247d2cb449e0a29f6f5d81518589 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/b885247d2cb449e0a29f6f5d81518589
I20250112 02:52:28.813808 1175 raft_consensus.cc:2267] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:28.813906 1171 ts_tablet_manager.cc:1939] T b885247d2cb449e0a29f6f5d81518589 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:28.814651 1175 ts_tablet_manager.cc:1905] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.814813 1563 tablet.cc:1721] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:28.816263 754 tablet_service.cc:1467] Processing CreateTablet for tablet b1d9fae290ec4281839991f9d40b105c (DEFAULT_TABLE table=default.Table_c1639450ddf94784a40b8cf28f3059c4 [id=93cb911ed4e045c994c9a06b10e2e4a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.817636 1177 tablet_bootstrap.cc:492] T cb724feccc7f4fbc846fb6fa61be3385 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:28.819315 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_e8e62bd2fffe4f50849f722dbece2a2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:28.819867 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8e62bd2fffe4f50849f722dbece2a2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:28.820094 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ead501d76bb1412e9839fb36b9911e6f consensus state has no leader
I20250112 02:52:28.820241 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 5601a9398c9a466582a98967c0e749e2
I20250112 02:52:28.820382 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 2052b4536f9745f0a785f330d706c1cf (DEFAULT_TABLE table=default.Table_096703f878244b8dac2dcf187fd23fd6 [id=e1779f88b20942eaa2c640ac24946fff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.821002 521 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_1d107b48a09e41d6bb3401b01a21ee78" } modify_external_catalogs: true
I20250112 02:52:28.821157 521 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_1d107b48a09e41d6bb3401b01a21ee78" } modify_external_catalogs: true
I20250112 02:52:28.821210 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2052b4536f9745f0a785f330d706c1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.821069 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 5601a9398c9a466582a98967c0e749e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:28.823588 1177 ts_tablet_manager.cc:1397] T cb724feccc7f4fbc846fb6fa61be3385 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.000s
I20250112 02:52:28.824173 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet b885247d2cb449e0a29f6f5d81518589 (table default.Table_edaa8b50dfa8463e99e05ab8d1d89bc3 [id=a6a18be9a78e4959b7d272c447f6fc25]) successfully deleted
I20250112 02:52:28.823908 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 5601a9398c9a466582a98967c0e749e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:28.824978 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1d9fae290ec4281839991f9d40b105c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.825191 1458 raft_consensus.cc:491] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.825371 1458 raft_consensus.cc:513] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:28.825700 1177 raft_consensus.cc:357] T cb724feccc7f4fbc846fb6fa61be3385 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.825910 1177 raft_consensus.cc:383] T cb724feccc7f4fbc846fb6fa61be3385 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.825994 1177 raft_consensus.cc:738] T cb724feccc7f4fbc846fb6fa61be3385 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.826184 1458 leader_election.cc:290] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:28.826220 1177 consensus_queue.cc:260] T cb724feccc7f4fbc846fb6fa61be3385 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.826515 899 tablet_service.cc:1467] Processing CreateTablet for tablet b1d9fae290ec4281839991f9d40b105c (DEFAULT_TABLE table=default.Table_c1639450ddf94784a40b8cf28f3059c4 [id=93cb911ed4e045c994c9a06b10e2e4a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.826848 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1d9fae290ec4281839991f9d40b105c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.826853 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c530aa0aa64e47ae27d61f341d2bd9" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:28.827497 855 leader_election.cc:343] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.827888 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet b1d9fae290ec4281839991f9d40b105c
I20250112 02:52:28.828729 895 tablet_service.cc:1514] Processing DeleteTablet for tablet b1d9fae290ec4281839991f9d40b105c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:28.828796 1177 ts_tablet_manager.cc:1428] T cb724feccc7f4fbc846fb6fa61be3385 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:28.829449 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet b1d9fae290ec4281839991f9d40b105c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:28.829416 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 6171b8ff2bfe41d598b0cb2535799a11 (DEFAULT_TABLE table=default.Table_e14b6c6d907942ebb7f8410e18d38ce2 [id=3097d5437937423e95b710c00473b6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.830330 1177 tablet_bootstrap.cc:492] T b2d8ea2ef18a46508371b4789ab6f4fa P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:28.830368 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c530aa0aa64e47ae27d61f341d2bd9" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:28.830386 1035 tablet_service.cc:1467] Processing CreateTablet for tablet b1d9fae290ec4281839991f9d40b105c (DEFAULT_TABLE table=default.Table_c1639450ddf94784a40b8cf28f3059c4 [id=93cb911ed4e045c994c9a06b10e2e4a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.830886 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1d9fae290ec4281839991f9d40b105c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.831964 1177 tablet_bootstrap.cc:654] T b2d8ea2ef18a46508371b4789ab6f4fa P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:28.832751 854 leader_election.cc:343] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.832794 759 tablet_service.cc:1467] Processing CreateTablet for tablet 79eb61cedd544c49869471eebe2fd237 (DEFAULT_TABLE table=default.Table_5adbd32e9a0c4736844af4c72b7787fd [id=3a629cb4a4104e30b0beb8f729ba0a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.832885 751 tablet_service.cc:1514] Processing DeleteTablet for tablet b1d9fae290ec4281839991f9d40b105c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:28.832883 854 leader_election.cc:304] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:28.833060 1171 tablet_replica.cc:331] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:28.833125 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79eb61cedd544c49869471eebe2fd237. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.833226 1171 raft_consensus.cc:2238] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.833389 1171 raft_consensus.cc:2267] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:28.833621 1458 raft_consensus.cc:2743] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:28.833844 1171 ts_tablet_manager.cc:1905] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.837989 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_06b4717d1d6d4b73b1488df2da81bbdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:28.838433 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06b4717d1d6d4b73b1488df2da81bbdc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:28.840144 1177 tablet_bootstrap.cc:492] T b2d8ea2ef18a46508371b4789ab6f4fa P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:28.840328 1177 ts_tablet_manager.cc:1397] T b2d8ea2ef18a46508371b4789ab6f4fa P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20250112 02:52:28.841063 1177 raft_consensus.cc:357] T b2d8ea2ef18a46508371b4789ab6f4fa P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.841287 1177 raft_consensus.cc:383] T b2d8ea2ef18a46508371b4789ab6f4fa P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.841413 1177 raft_consensus.cc:738] T b2d8ea2ef18a46508371b4789ab6f4fa P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.841410 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_3fec5f647ce141459d216dca71e5eb87" } new_table_name: "default.Table_5a593457b27a46dabbb7a2bfc8332e9c" modify_external_catalogs: true
I20250112 02:52:28.841503 517 catalog_manager.cc:6071] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190] alter complete (version 2)
I20250112 02:52:28.841696 1177 consensus_queue.cc:260] T b2d8ea2ef18a46508371b4789ab6f4fa P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.844749 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_2750897a0a504c2ca90590075d4cf54e" } new_table_name: "default.Table_26dfe346379948b39f90b31653c2a463" modify_external_catalogs: true
I20250112 02:52:28.845721 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_f0fb142b9aab490bb067d3cf64dc47b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:28.845922 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_9c10574f7c7d435890eb9a24af23bcb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:28.846019 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_b0d877b624604ddc93c56bec6e05261f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:28.846264 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0fb142b9aab490bb067d3cf64dc47b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:28.846545 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0d877b624604ddc93c56bec6e05261f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:28.846602 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_d0a145ff8f4d4e0484e015fb4d18860c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:28.847036 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0a145ff8f4d4e0484e015fb4d18860c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:28.846292 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c10574f7c7d435890eb9a24af23bcb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:28.848004 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_254eb676d963466fb32f0df6af436598" } new_table_name: "default.Table_ba2bcb3825c0471a9bff813594d354f9" modify_external_catalogs: true
I20250112 02:52:28.849323 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_29a6037a21724b5f9f0e0ee6a598a3dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:28.849797 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29a6037a21724b5f9f0e0ee6a598a3dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:28.852039 1177 ts_tablet_manager.cc:1428] T b2d8ea2ef18a46508371b4789ab6f4fa P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250112 02:52:28.852304 1177 tablet_bootstrap.cc:492] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:28.853718 1177 tablet_bootstrap.cc:654] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:28.854018 1564 tablet.cc:1721] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:28.857868 1171 ts_tablet_manager.cc:1918] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:28.857987 1171 log.cc:1198] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/5601a9398c9a466582a98967c0e749e2
I20250112 02:52:28.858340 1171 ts_tablet_manager.cc:1939] T 5601a9398c9a466582a98967c0e749e2 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:28.859229 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 5601a9398c9a466582a98967c0e749e2 (table default.Table_3b1c87a5169e49b68fdf3d5c662a27bc [id=ed3672fa460a4aa2ae1d3b643ade43e6]) successfully deleted
W20250112 02:52:28.859884 513 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b1d9fae290ec4281839991f9d40b105c because the tablet was not found. No further retry: Not found: Tablet not found: b1d9fae290ec4281839991f9d40b105c
I20250112 02:52:28.866638 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 79eb61cedd544c49869471eebe2fd237 (DEFAULT_TABLE table=default.Table_5adbd32e9a0c4736844af4c72b7787fd [id=3a629cb4a4104e30b0beb8f729ba0a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.867228 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6171b8ff2bfe41d598b0cb2535799a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.869628 1175 ts_tablet_manager.cc:1918] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:28.873214 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 5601a9398c9a466582a98967c0e749e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:28.873410 1175 log.cc:1198] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/b885247d2cb449e0a29f6f5d81518589
I20250112 02:52:28.875500 1175 ts_tablet_manager.cc:1939] T b885247d2cb449e0a29f6f5d81518589 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:28.876194 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 258) took 2055 ms. Trace:
I20250112 02:52:28.876746 1177 tablet_bootstrap.cc:492] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:28.876870 1177 ts_tablet_manager.cc:1397] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.011s
I20250112 02:52:28.876883 1175 rpcz_store.cc:276] 0112 02:52:26.820630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.821004 (+ 374us) service_pool.cc:225] Handling call
0112 02:52:28.812693 (+1991689us) ts_tablet_manager.cc:1145] Deleting tablet b885247d2cb449e0a29f6f5d81518589
0112 02:52:28.812701 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:28.862997 (+ 50296us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:28.869508 (+ 6511us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:28.876162 (+ 6654us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1990565}
I20250112 02:52:28.877446 1543 tablet.cc:1721] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:28.878316 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b
I20250112 02:52:28.880149 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_5e6ad2a2040d41b381bb2653c6f691ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:28.880694 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e6ad2a2040d41b381bb2653c6f691ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:28.880620 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_d45a3d1878fc47d5917bf66c670c5578"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:28.881068 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d45a3d1878fc47d5917bf66c670c5578 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:28.882570 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:28.882647 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet b885247d2cb449e0a29f6f5d81518589 (table default.Table_edaa8b50dfa8463e99e05ab8d1d89bc3 [id=a6a18be9a78e4959b7d272c447f6fc25]) successfully deleted
I20250112 02:52:28.882782 1175 tablet_replica.cc:331] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:28.882951 1175 raft_consensus.cc:2238] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.883106 1175 raft_consensus.cc:2267] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:28.883247 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9af55dab202444358ea1a0674d37946d consensus state has no leader
I20250112 02:52:28.883306 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:28.883553 757 tablet_service.cc:1514] Processing DeleteTablet for tablet cb724feccc7f4fbc846fb6fa61be3385 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:28.883728 753 tablet_service.cc:1514] Processing DeleteTablet for tablet b2d8ea2ef18a46508371b4789ab6f4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:28.884948 1171 tablet_replica.cc:331] T cb724feccc7f4fbc846fb6fa61be3385 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:28.885110 1171 raft_consensus.cc:2238] T cb724feccc7f4fbc846fb6fa61be3385 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.885282 1171 raft_consensus.cc:2267] T cb724feccc7f4fbc846fb6fa61be3385 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:28.886734 1175 ts_tablet_manager.cc:1905] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.887602 1171 ts_tablet_manager.cc:1905] T cb724feccc7f4fbc846fb6fa61be3385 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.889690 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:28.892904 1171 ts_tablet_manager.cc:1918] T cb724feccc7f4fbc846fb6fa61be3385 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:28.893007 1171 log.cc:1198] T cb724feccc7f4fbc846fb6fa61be3385 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/cb724feccc7f4fbc846fb6fa61be3385
I20250112 02:52:28.893374 1171 ts_tablet_manager.cc:1939] T cb724feccc7f4fbc846fb6fa61be3385 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:28.894379 1171 tablet_replica.cc:331] T b2d8ea2ef18a46508371b4789ab6f4fa P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:28.894598 1171 raft_consensus.cc:2238] T b2d8ea2ef18a46508371b4789ab6f4fa P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.894767 1171 raft_consensus.cc:2267] T b2d8ea2ef18a46508371b4789ab6f4fa P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:28.895382 1173 tablet_bootstrap.cc:492] T 54033a747d5a49b0a1c16f138158f596 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:28.895571 1171 ts_tablet_manager.cc:1905] T b2d8ea2ef18a46508371b4789ab6f4fa P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.895682 1173 ts_tablet_manager.cc:1397] T 54033a747d5a49b0a1c16f138158f596 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.243s user 0.005s sys 0.027s
I20250112 02:52:28.896042 1172 tablet_replica.cc:331] T 99715f37fbef47c6860fa1d95e350f20 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:28.900477 1504 catalog_manager.cc:1260] Loaded cluster ID: 273ff2f1d16946fcac5345b54ec7471c
I20250112 02:52:28.900578 1504 catalog_manager.cc:1553] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304: loading cluster ID for follower catalog manager: success
I20250112 02:52:28.877524 1177 raft_consensus.cc:357] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.901387 1177 raft_consensus.cc:383] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.901484 1177 raft_consensus.cc:738] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.901682 1177 consensus_queue.cc:260] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.902238 1177 ts_tablet_manager.cc:1428] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.025s user 0.000s sys 0.001s
I20250112 02:52:28.902787 1177 tablet_bootstrap.cc:492] T aba1744138fe42ae9d097db01cac083d P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:28.904758 1177 tablet_bootstrap.cc:654] T aba1744138fe42ae9d097db01cac083d P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:28.905557 1171 ts_tablet_manager.cc:1918] T b2d8ea2ef18a46508371b4789ab6f4fa P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:28.905685 1171 log.cc:1198] T b2d8ea2ef18a46508371b4789ab6f4fa P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/b2d8ea2ef18a46508371b4789ab6f4fa
I20250112 02:52:28.905977 1171 ts_tablet_manager.cc:1939] T b2d8ea2ef18a46508371b4789ab6f4fa P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:28.913720 1172 raft_consensus.cc:2238] T 99715f37fbef47c6860fa1d95e350f20 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.913919 1172 raft_consensus.cc:2267] T 99715f37fbef47c6860fa1d95e350f20 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:28.914456 1172 ts_tablet_manager.cc:1905] T 99715f37fbef47c6860fa1d95e350f20 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.919272 1537 raft_consensus.cc:491] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.919435 1537 raft_consensus.cc:513] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.920073 1537 leader_election.cc:290] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:28.921267 989 leader_election.cc:343] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: fb4c004397ee480d897e8b125d248829
I20250112 02:52:28.927415 1173 raft_consensus.cc:357] T 54033a747d5a49b0a1c16f138158f596 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.927649 1173 raft_consensus.cc:383] T 54033a747d5a49b0a1c16f138158f596 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.927716 1173 raft_consensus.cc:738] T 54033a747d5a49b0a1c16f138158f596 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.927923 1173 consensus_queue.cc:260] T 54033a747d5a49b0a1c16f138158f596 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.928380 1173 ts_tablet_manager.cc:1428] T 54033a747d5a49b0a1c16f138158f596 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.033s user 0.001s sys 0.000s
I20250112 02:52:28.928685 1504 catalog_manager.cc:1575] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304: acquiring CA information for follower catalog manager: success
W20250112 02:52:28.929119 991 leader_election.cc:343] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: fb4c004397ee480d897e8b125d248829
I20250112 02:52:28.929222 991 leader_election.cc:304] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:28.929289 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79eb61cedd544c49869471eebe2fd237. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.930305 1537 raft_consensus.cc:2743] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:28.932930 1458 raft_consensus.cc:491] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:28.933084 1458 raft_consensus.cc:513] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:28.933290 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fccff64af8ca4400afc116db98918119 consensus state has no leader
I20250112 02:52:28.933720 1458 leader_election.cc:290] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:28.934070 899 tablet_service.cc:1467] Processing CreateTablet for tablet 79eb61cedd544c49869471eebe2fd237 (DEFAULT_TABLE table=default.Table_5adbd32e9a0c4736844af4c72b7787fd [id=3a629cb4a4104e30b0beb8f729ba0a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.934206 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cb7e7a41634cf1bdb978ac9d2befcd" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:28.934420 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79eb61cedd544c49869471eebe2fd237. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:28.934648 855 leader_election.cc:343] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.936082 1181 tablet_bootstrap.cc:492] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:28.936203 1181 ts_tablet_manager.cc:1397] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.205s user 0.002s sys 0.005s
I20250112 02:52:28.936451 1177 tablet_bootstrap.cc:492] T aba1744138fe42ae9d097db01cac083d P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:28.936582 1177 ts_tablet_manager.cc:1397] T aba1744138fe42ae9d097db01cac083d P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.024s
I20250112 02:52:28.936700 1181 raft_consensus.cc:357] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.936838 1181 raft_consensus.cc:383] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.936882 1181 raft_consensus.cc:738] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.937044 1181 consensus_queue.cc:260] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.937136 1177 raft_consensus.cc:357] T aba1744138fe42ae9d097db01cac083d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.937331 1177 raft_consensus.cc:383] T aba1744138fe42ae9d097db01cac083d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:28.937388 1177 raft_consensus.cc:738] T aba1744138fe42ae9d097db01cac083d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:28.937429 1181 ts_tablet_manager.cc:1428] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:28.937685 1181 tablet_bootstrap.cc:492] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:28.937598 1177 consensus_queue.cc:260] T aba1744138fe42ae9d097db01cac083d P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:28.938023 1177 ts_tablet_manager.cc:1428] T aba1744138fe42ae9d097db01cac083d P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 02:52:28.938264 1177 tablet_bootstrap.cc:492] T e9fc9a4f1e8944abb76dad6c2ad53d35 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:28.939353 1181 tablet_bootstrap.cc:654] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:28.939520 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b already in progress: creating tablet
I20250112 02:52:28.939766 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet cb724feccc7f4fbc846fb6fa61be3385 (table default.Table_43048dfe7bf24df5a7254bdb6854101e [id=81a09e598ee34508aa88abfc6b44a2bd]) successfully deleted
I20250112 02:52:28.940068 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet b2d8ea2ef18a46508371b4789ab6f4fa (table default.Table_12a25f7a620047b69f34be9eeae64dd2 [id=87c1bff49b4346599bf98166ce8672d4]) successfully deleted
I20250112 02:52:28.940229 1177 tablet_bootstrap.cc:654] T e9fc9a4f1e8944abb76dad6c2ad53d35 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:28.940907 754 tablet_service.cc:1467] Processing CreateTablet for tablet 37862023095644bab2403576e8cb8c58 (DEFAULT_TABLE table=default.Table_8034217cf2a247d5af40b1123517157f [id=01ca9afae4dc47499802f76b7ee93bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.944088 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37862023095644bab2403576e8cb8c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.947073 893 tablet_service.cc:1467] Processing CreateTablet for tablet 37862023095644bab2403576e8cb8c58 (DEFAULT_TABLE table=default.Table_8034217cf2a247d5af40b1123517157f [id=01ca9afae4dc47499802f76b7ee93bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.947438 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37862023095644bab2403576e8cb8c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.948668 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cb7e7a41634cf1bdb978ac9d2befcd" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:28.949422 759 tablet_service.cc:1514] Processing DeleteTablet for tablet b2d8ea2ef18a46508371b4789ab6f4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:28.951763 854 leader_election.cc:343] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:28.951905 854 leader_election.cc:304] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:28.952245 1458 raft_consensus.cc:2743] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:28.952804 1173 tablet_bootstrap.cc:492] T 9427951452104a7db765437493b21e90 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:28.956835 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 37862023095644bab2403576e8cb8c58 (DEFAULT_TABLE table=default.Table_8034217cf2a247d5af40b1123517157f [id=01ca9afae4dc47499802f76b7ee93bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:28.957136 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37862023095644bab2403576e8cb8c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:28.963951 1173 tablet_bootstrap.cc:654] T 9427951452104a7db765437493b21e90 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:28.965736 1175 ts_tablet_manager.cc:1918] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:28.965852 1175 log.cc:1198] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/c7c530aa0aa64e47ae27d61f341d2bd9
I20250112 02:52:28.966137 1175 ts_tablet_manager.cc:1939] T c7c530aa0aa64e47ae27d61f341d2bd9 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:28.966691 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 261) took 2110 ms. Trace:
I20250112 02:52:28.966770 1175 rpcz_store.cc:276] 0112 02:52:26.856036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.856148 (+ 112us) service_pool.cc:225] Handling call
0112 02:52:28.882729 (+2026581us) ts_tablet_manager.cc:1145] Deleting tablet c7c530aa0aa64e47ae27d61f341d2bd9
0112 02:52:28.882738 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:28.890190 (+ 7452us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:28.965666 (+ 75476us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:28.966657 (+ 991us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2025542}
I20250112 02:52:28.973404 525 master_service.cc:946] ReplaceTablet: received request to replace tablet 49bb6d20cfa0453e90e048825114525e from {username='slave'} at 127.0.0.1:40934
W20250112 02:52:28.973595 513 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b2d8ea2ef18a46508371b4789ab6f4fa because the tablet was not found. No further retry: Not found: Tablet not found: b2d8ea2ef18a46508371b4789ab6f4fa
I20250112 02:52:28.974460 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_e6b4b27b8ab44edcb86d1e93ec94ac0a" } new_table_name: "default.Table_eaf6d6a8cdc84bed8248e8af835d85b0" modify_external_catalogs: true
I20250112 02:52:28.976704 522 master_service.cc:946] ReplaceTablet: received request to replace tablet b6dc79f4d1194f81963a8094ed6b0d2e from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:28.977964 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_0f1762619e9543018fd2df57adb6367e" } modify_external_catalogs: true
I20250112 02:52:28.978909 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_0f1762619e9543018fd2df57adb6367e" } modify_external_catalogs: true
I20250112 02:52:28.980255 523 master_service.cc:946] ReplaceTablet: received request to replace tablet b6dc79f4d1194f81963a8094ed6b0d2e from {username='slave'} at 127.0.0.1:40934
W20250112 02:52:28.983412 855 consensus_peers.cc:487] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5601a9398c9a466582a98967c0e749e2. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:28.984939 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet c7c530aa0aa64e47ae27d61f341d2bd9 (table default.Table_247a821819994de6adac6fabd06c3a1f [id=60486b9e81254d74b8e08a1cadde13bd]) successfully deleted
I20250112 02:52:28.988648 525 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 09d06e2e8c13464db8cf801a292bdc99 (table default.Table_0bdc5f49d4ff4d6fb1290bc2dc0f70d9 [id=e3bb489c53454222a79702b798c0c2f1]). Expected version 2 got 0
I20250112 02:52:28.990293 754 tablet_service.cc:1514] Processing DeleteTablet for tablet aba1744138fe42ae9d097db01cac083d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:28.990542 1171 tablet_replica.cc:331] T aba1744138fe42ae9d097db01cac083d P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:28.990672 893 tablet_service.cc:1514] Processing DeleteTablet for tablet aadb2ab1d2024f889841ac91e065e064 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:28.990742 1171 raft_consensus.cc:2238] T aba1744138fe42ae9d097db01cac083d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.990914 1171 raft_consensus.cc:2267] T aba1744138fe42ae9d097db01cac083d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:28.990975 525 master_service.cc:946] ReplaceTablet: received request to replace tablet aadb2ab1d2024f889841ac91e065e064 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:28.991520 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_b2b3c738a24f451eac39fb1a6399f37a" } modify_external_catalogs: true
I20250112 02:52:28.991624 1171 ts_tablet_manager.cc:1905] T aba1744138fe42ae9d097db01cac083d P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.994473 1172 ts_tablet_manager.cc:1918] T 99715f37fbef47c6860fa1d95e350f20 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:28.994611 1172 log.cc:1198] T 99715f37fbef47c6860fa1d95e350f20 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/99715f37fbef47c6860fa1d95e350f20
I20250112 02:52:28.994899 1172 ts_tablet_manager.cc:1939] T 99715f37fbef47c6860fa1d95e350f20 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:28.995297 1172 tablet_replica.cc:331] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:28.995488 1172 raft_consensus.cc:2238] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:28.995649 1172 raft_consensus.cc:2267] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:28.996119 1172 ts_tablet_manager.cc:1905] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:28.996623 1171 ts_tablet_manager.cc:1918] T aba1744138fe42ae9d097db01cac083d P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:28.996737 1171 log.cc:1198] T aba1744138fe42ae9d097db01cac083d P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/aba1744138fe42ae9d097db01cac083d
I20250112 02:52:28.997061 1171 ts_tablet_manager.cc:1939] T aba1744138fe42ae9d097db01cac083d P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:28.999737 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 99715f37fbef47c6860fa1d95e350f20 (table default.Table_b56246b266524f2eb968fe7df99ce8b2 [id=d3a0fac8c9234ccc92b4c929c973693d]) successfully deleted
I20250112 02:52:29.000667 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 263) took 2144 ms. Trace:
I20250112 02:52:29.000777 1175 rpcz_store.cc:276] 0112 02:52:26.856270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.856306 (+ 36us) service_pool.cc:225] Handling call
0112 02:52:29.000522 (+2144216us) ts_tablet_manager.cc:1145] Deleting tablet b9199b0d2b924a79938a979df4196bb4
0112 02:52:29.000529 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:29.000619 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2138427}
I20250112 02:52:29.001806 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 8c821ea3ca4946c290c17135366e0cac
I20250112 02:52:29.002738 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c821ea3ca4946c290c17135366e0cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:29.003777 1181 tablet_bootstrap.cc:492] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:29.003908 1181 ts_tablet_manager.cc:1397] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.005s
I20250112 02:52:29.005160 1181 raft_consensus.cc:357] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.005355 1181 raft_consensus.cc:383] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.005439 1181 raft_consensus.cc:738] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.005654 1181 consensus_queue.cc:260] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.006206 1172 ts_tablet_manager.cc:1918] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:29.006350 1172 log.cc:1198] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/d6ee0e79dbb34b1ba9dcd39d36aa2ae7
I20250112 02:52:29.006738 1172 ts_tablet_manager.cc:1939] T d6ee0e79dbb34b1ba9dcd39d36aa2ae7 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:29.007620 1172 tablet_replica.cc:331] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:29.007845 1172 raft_consensus.cc:2238] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.008047 1172 raft_consensus.cc:2267] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.011709 1181 ts_tablet_manager.cc:1428] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250112 02:52:29.017204 1172 ts_tablet_manager.cc:1905] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.023921 1504 catalog_manager.cc:1603] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 02:52:29.026021 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_5a593457b27a46dabbb7a2bfc8332e9c" } new_table_name: "default.Table_15bcb2c0f2a74458a1bed231fd3437fc" modify_external_catalogs: true
I20250112 02:52:29.028204 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_6671f628f89b428a83eca77e9ca26516"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:29.028988 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet aba1744138fe42ae9d097db01cac083d (table default.Table_fb7e727aeed44203b2bb7e90a844e44b [id=587926735ced4c17b747f5c1fbee9f8d]) successfully deleted
I20250112 02:52:29.029661 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c821ea3ca4946c290c17135366e0cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:29.030826 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet b9199b0d2b924a79938a979df4196bb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9199b0d2b924a79938a979df4196bb4 already in progress: creating tablet
W20250112 02:52:29.031152 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6671f628f89b428a83eca77e9ca26516 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:29.031780 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c821ea3ca4946c290c17135366e0cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:29.033401 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a6650515ea540a8bbf21e2f88363360 has no consensus state
I20250112 02:52:29.034215 1181 tablet_bootstrap.cc:492] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:29.034143 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_b2b3c738a24f451eac39fb1a6399f37a" } modify_external_catalogs: true
I20250112 02:52:29.034410 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet b6dc79f4d1194f81963a8094ed6b0d2e
I20250112 02:52:29.035032 522 catalog_manager.cc:6596] ReplaceTablet: tablet b6dc79f4d1194f81963a8094ed6b0d2e deleted and replaced by tablet b7340a7a916c4b2fab32e0ba2d0ce322
I20250112 02:52:29.035218 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet d6ee0e79dbb34b1ba9dcd39d36aa2ae7 (table default.Table_c360a95859e5448f9af5f90629a898e4 [id=08b0c7e4e0c54ec4a215a390e83a1cf6]) successfully deleted
I20250112 02:52:29.035785 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet b6dc79f4d1194f81963a8094ed6b0d2e with delete_type TABLET_DATA_DELETED (replaced by tablet b7340a7a916c4b2fab32e0ba2d0ce322) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:29.036995 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet b6dc79f4d1194f81963a8094ed6b0d2e with delete_type TABLET_DATA_DELETED (replaced by tablet b7340a7a916c4b2fab32e0ba2d0ce322) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:29.037338 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_8e00e020edc14465ad0e935be514d03d" } modify_external_catalogs: true
I20250112 02:52:29.037429 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 262) took 2181 ms. Trace:
I20250112 02:52:29.037485 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_8e00e020edc14465ad0e935be514d03d" } modify_external_catalogs: true
I20250112 02:52:29.037499 1175 rpcz_store.cc:276] 0112 02:52:26.856167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.856280 (+ 113us) service_pool.cc:225] Handling call
0112 02:52:29.037290 (+2181010us) ts_tablet_manager.cc:1145] Deleting tablet 4b599e983ec04f53a440d1ab828d12ab
0112 02:52:29.037300 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:29.037391 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2171294}
I20250112 02:52:29.038324 1175 tablet_replica.cc:331] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:29.038738 1175 raft_consensus.cc:2238] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.039132 754 tablet_service.cc:1514] Processing DeleteTablet for tablet b6dc79f4d1194f81963a8094ed6b0d2e with delete_type TABLET_DATA_DELETED (replaced by tablet b7340a7a916c4b2fab32e0ba2d0ce322) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:29.039173 1175 raft_consensus.cc:2267] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.039400 1171 tablet_replica.cc:331] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:29.039615 1171 raft_consensus.cc:2238] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:29.039700 893 tablet_service.cc:1514] Processing DeleteTablet for tablet b6dc79f4d1194f81963a8094ed6b0d2e with delete_type TABLET_DATA_DELETED (replaced by tablet b7340a7a916c4b2fab32e0ba2d0ce322) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:29.039906 1171 raft_consensus.cc:2267] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:29.040030 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c73f2fa452b84be398476b8aba6c9a08 consensus state has no leader
I20250112 02:52:29.040463 1172 ts_tablet_manager.cc:1918] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:29.040601 1172 log.cc:1198] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/fb4c004397ee480d897e8b125d248829
I20250112 02:52:29.040807 1181 tablet_bootstrap.cc:654] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.040942 1172 ts_tablet_manager.cc:1939] T fb4c004397ee480d897e8b125d248829 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:29.041072 1171 ts_tablet_manager.cc:1905] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.041447 1172 tablet_replica.cc:331] T 328e307a4b35445f92cbfab24f2ada48 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:29.041631 1172 raft_consensus.cc:2238] T 328e307a4b35445f92cbfab24f2ada48 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.041823 1172 raft_consensus.cc:2267] T 328e307a4b35445f92cbfab24f2ada48 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.044481 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_2a2995d7964346c08fcbff96b4cc3ca5" } modify_external_catalogs: true
I20250112 02:52:29.044639 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_2a2995d7964346c08fcbff96b4cc3ca5" } modify_external_catalogs: true
W20250112 02:52:29.046049 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 8c821ea3ca4946c290c17135366e0cac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c821ea3ca4946c290c17135366e0cac already in progress: creating tablet
W20250112 02:52:29.046258 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet bb217afc08304c0ea35dee3120347ba1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb217afc08304c0ea35dee3120347ba1 already in progress: creating tablet
W20250112 02:52:29.047854 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 4b599e983ec04f53a440d1ab828d12ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b599e983ec04f53a440d1ab828d12ab already in progress: creating tablet
I20250112 02:52:29.048527 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet fb4c004397ee480d897e8b125d248829 (table default.Table_091c87de29164fc784158659eeaf545a [id=b8c138bea585437c83ee0ed94c008284]) successfully deleted
I20250112 02:52:29.050379 1171 ts_tablet_manager.cc:1918] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:29.050508 1171 log.cc:1198] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/b6dc79f4d1194f81963a8094ed6b0d2e
I20250112 02:52:29.050933 1171 ts_tablet_manager.cc:1939] T b6dc79f4d1194f81963a8094ed6b0d2e P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:29.052084 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet b6dc79f4d1194f81963a8094ed6b0d2e (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]) successfully deleted
I20250112 02:52:29.053782 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_2db060823dc54c289c2595ad2684d694"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:29.054034 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c686fdc8896463c81c9f7c5f53a3da2 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c03b5e612d64314b210c11686fdfa2a) from {username='slave'} at 127.0.0.1:51456
W20250112 02:52:29.054579 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2db060823dc54c289c2595ad2684d694 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:29.056739 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 54033a747d5a49b0a1c16f138158f596 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:29.060554 522 master_service.cc:946] ReplaceTablet: received request to replace tablet d45fce86e02f46d4a57af718fa2a62c8 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:29.064069 1175 ts_tablet_manager.cc:1905] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:29.064522 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fccff64af8ca4400afc116db98918119 consensus state has no leader
I20250112 02:52:29.065160 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet daef3a58605e40978f7389698b22265d
I20250112 02:52:29.066610 893 tablet_service.cc:1514] Processing DeleteTablet for tablet daef3a58605e40978f7389698b22265d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:29.066787 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet daef3a58605e40978f7389698b22265d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:29.067195 754 tablet_service.cc:1514] Processing DeleteTablet for tablet daef3a58605e40978f7389698b22265d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:29.067811 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet daef3a58605e40978f7389698b22265d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet daef3a58605e40978f7389698b22265d already in progress: creating tablet
I20250112 02:52:29.072151 1173 tablet_bootstrap.cc:492] T 9427951452104a7db765437493b21e90 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:29.072286 1173 ts_tablet_manager.cc:1397] T 9427951452104a7db765437493b21e90 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.124s user 0.021s sys 0.021s
I20250112 02:52:29.074472 1172 ts_tablet_manager.cc:1905] T 328e307a4b35445f92cbfab24f2ada48 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.074524 1173 raft_consensus.cc:357] T 9427951452104a7db765437493b21e90 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.074679 1173 raft_consensus.cc:383] T 9427951452104a7db765437493b21e90 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.074781 1173 raft_consensus.cc:738] T 9427951452104a7db765437493b21e90 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.074990 1173 consensus_queue.cc:260] T 9427951452104a7db765437493b21e90 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.075691 1173 ts_tablet_manager.cc:1428] T 9427951452104a7db765437493b21e90 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250112 02:52:29.076056 1173 tablet_bootstrap.cc:492] T b885247d2cb449e0a29f6f5d81518589 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:29.077500 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet fe614eaeb793473d9b2da206b7345562
I20250112 02:52:29.078275 893 tablet_service.cc:1514] Processing DeleteTablet for tablet fe614eaeb793473d9b2da206b7345562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:29.078325 754 tablet_service.cc:1514] Processing DeleteTablet for tablet fe614eaeb793473d9b2da206b7345562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:29.078807 1173 tablet_bootstrap.cc:654] T b885247d2cb449e0a29f6f5d81518589 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:29.079201 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet fe614eaeb793473d9b2da206b7345562 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe614eaeb793473d9b2da206b7345562 already in progress: creating tablet
I20250112 02:52:29.080895 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 06edfca60eed4dd3a2dee7b662b0970c
I20250112 02:52:29.081508 1172 ts_tablet_manager.cc:1918] T 328e307a4b35445f92cbfab24f2ada48 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:29.081609 1172 log.cc:1198] T 328e307a4b35445f92cbfab24f2ada48 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/328e307a4b35445f92cbfab24f2ada48
I20250112 02:52:29.081915 1172 ts_tablet_manager.cc:1939] T 328e307a4b35445f92cbfab24f2ada48 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:29.081966 1175 ts_tablet_manager.cc:1918] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:29.082077 1175 log.cc:1198] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/aba1744138fe42ae9d097db01cac083d
I20250112 02:52:29.082448 1175 ts_tablet_manager.cc:1939] T aba1744138fe42ae9d097db01cac083d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:29.083235 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 269) took 2083 ms. Trace:
I20250112 02:52:29.083359 1175 rpcz_store.cc:276] 0112 02:52:26.999698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:26.999762 (+ 64us) service_pool.cc:225] Handling call
0112 02:52:29.038258 (+2038496us) ts_tablet_manager.cc:1145] Deleting tablet aba1744138fe42ae9d097db01cac083d
0112 02:52:29.038266 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:29.074023 (+ 35757us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:29.081844 (+ 7821us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:29.083197 (+ 1353us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2038232}
I20250112 02:52:29.083745 1175 tablet_replica.cc:331] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:29.083956 1175 raft_consensus.cc:2238] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.084105 1175 raft_consensus.cc:2267] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.084681 1172 tablet_replica.cc:331] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:29.084674 1175 ts_tablet_manager.cc:1905] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.084844 1172 raft_consensus.cc:2238] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.084995 1172 raft_consensus.cc:2267] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.085726 1172 ts_tablet_manager.cc:1905] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.087353 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet aba1744138fe42ae9d097db01cac083d (table default.Table_fb7e727aeed44203b2bb7e90a844e44b [id=587926735ced4c17b747f5c1fbee9f8d]) successfully deleted
I20250112 02:52:29.087594 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 06edfca60eed4dd3a2dee7b662b0970c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:29.088689 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_2e35ffa5571d416d9b68443869009ab4" } new_table_name: "default.Table_83212f303f0b4413aa6c62887da9dd34" modify_external_catalogs: true
I20250112 02:52:29.088840 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_fc264b6a06584c7ea757f5d18e05e2bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:29.089105 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 06edfca60eed4dd3a2dee7b662b0970c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:29.089043 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_6e77252e817f43399c9b3f5b221795ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:29.089329 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc264b6a06584c7ea757f5d18e05e2bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:29.089432 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e77252e817f43399c9b3f5b221795ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:29.089639 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 06edfca60eed4dd3a2dee7b662b0970c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06edfca60eed4dd3a2dee7b662b0970c already in progress: creating tablet
I20250112 02:52:29.089999 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_613140584042445b9d71a3d05cee62c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:29.090444 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_613140584042445b9d71a3d05cee62c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:29.091578 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_7921b99e24f84cceb58128f18925ad49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:29.091835 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_218fd160176b4f55937d334142f2ecbd" } modify_external_catalogs: true
W20250112 02:52:29.091959 513 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c73f2fa452b84be398476b8aba6c9a08 consensus state has no leader
W20250112 02:52:29.092024 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7921b99e24f84cceb58128f18925ad49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:29.091976 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_218fd160176b4f55937d334142f2ecbd" } modify_external_catalogs: true
I20250112 02:52:29.092286 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_b8af9fadb7b64c07afb49c07cf3ca5f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:29.092671 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8af9fadb7b64c07afb49c07cf3ca5f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:29.093169 1177 tablet_bootstrap.cc:492] T e9fc9a4f1e8944abb76dad6c2ad53d35 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:29.093290 1177 ts_tablet_manager.cc:1397] T e9fc9a4f1e8944abb76dad6c2ad53d35 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.155s user 0.037s sys 0.062s
I20250112 02:52:29.093955 1177 raft_consensus.cc:357] T e9fc9a4f1e8944abb76dad6c2ad53d35 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.094125 1177 raft_consensus.cc:383] T e9fc9a4f1e8944abb76dad6c2ad53d35 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.094216 1177 raft_consensus.cc:738] T e9fc9a4f1e8944abb76dad6c2ad53d35 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.094436 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 328e307a4b35445f92cbfab24f2ada48 (table default.Table_fc5d684c7f2049e5b2976823cfb242ae [id=c84a7bdac0ed4c2ea7c0599d5f1ca9c1]) successfully deleted
I20250112 02:52:29.094496 1177 consensus_queue.cc:260] T e9fc9a4f1e8944abb76dad6c2ad53d35 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:29.094635 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 65f22a7b1b804661af3b7e785a833739 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65f22a7b1b804661af3b7e785a833739 already in progress: creating tablet
W20250112 02:52:29.094808 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 70715db0394d46d9b6b289160b00be7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70715db0394d46d9b6b289160b00be7d already in progress: creating tablet
W20250112 02:52:29.094954 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b1d9fae290ec4281839991f9d40b105c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1d9fae290ec4281839991f9d40b105c already in progress: creating tablet
I20250112 02:52:29.094971 1177 ts_tablet_manager.cc:1428] T e9fc9a4f1e8944abb76dad6c2ad53d35 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:29.095285 1177 tablet_bootstrap.cc:492] T 6b521522c4784d2ab4056268823f1099 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:29.095883 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 5601a9398c9a466582a98967c0e749e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5601a9398c9a466582a98967c0e749e2 already in progress: creating tablet
W20250112 02:52:29.096113 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b already in progress: creating tablet
W20250112 02:52:29.096282 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 8c821ea3ca4946c290c17135366e0cac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c821ea3ca4946c290c17135366e0cac already in progress: creating tablet
I20250112 02:52:29.096853 1177 tablet_bootstrap.cc:654] T 6b521522c4784d2ab4056268823f1099 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.097793 754 tablet_service.cc:1514] Processing DeleteTablet for tablet e9fc9a4f1e8944abb76dad6c2ad53d35 with delete_type TABLET_DATA_DELETED (replaced by tablet 5cc414d6420c41839621ab858067548b) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:29.099931 1172 ts_tablet_manager.cc:1918] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:29.100126 1172 log.cc:1198] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/b6dc79f4d1194f81963a8094ed6b0d2e
I20250112 02:52:29.100248 1171 tablet_replica.cc:331] T e9fc9a4f1e8944abb76dad6c2ad53d35 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:29.100442 1171 raft_consensus.cc:2238] T e9fc9a4f1e8944abb76dad6c2ad53d35 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.100653 1171 raft_consensus.cc:2267] T e9fc9a4f1e8944abb76dad6c2ad53d35 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.101236 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 06edfca60eed4dd3a2dee7b662b0970c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:29.101347 1172 ts_tablet_manager.cc:1939] T b6dc79f4d1194f81963a8094ed6b0d2e P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:29.101548 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet fe614eaeb793473d9b2da206b7345562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:29.102293 1173 tablet_bootstrap.cc:492] T b885247d2cb449e0a29f6f5d81518589 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:29.102448 1173 ts_tablet_manager.cc:1397] T b885247d2cb449e0a29f6f5d81518589 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.002s
I20250112 02:52:29.102571 1172 tablet_replica.cc:331] T 54033a747d5a49b0a1c16f138158f596 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:29.102727 1172 raft_consensus.cc:2238] T 54033a747d5a49b0a1c16f138158f596 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.102881 1172 raft_consensus.cc:2267] T 54033a747d5a49b0a1c16f138158f596 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.103374 1172 ts_tablet_manager.cc:1905] T 54033a747d5a49b0a1c16f138158f596 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.103328 1173 raft_consensus.cc:357] T b885247d2cb449e0a29f6f5d81518589 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:29.103551 1173 raft_consensus.cc:383] T b885247d2cb449e0a29f6f5d81518589 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.103641 1173 raft_consensus.cc:738] T b885247d2cb449e0a29f6f5d81518589 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.103914 1173 consensus_queue.cc:260] T b885247d2cb449e0a29f6f5d81518589 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:29.104475 1173 ts_tablet_manager.cc:1428] T b885247d2cb449e0a29f6f5d81518589 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:29.104763 1173 tablet_bootstrap.cc:492] T 5601a9398c9a466582a98967c0e749e2 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:29.106570 1173 tablet_bootstrap.cc:654] T 5601a9398c9a466582a98967c0e749e2 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.110827 1171 ts_tablet_manager.cc:1905] T e9fc9a4f1e8944abb76dad6c2ad53d35 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.113678 1177 tablet_bootstrap.cc:492] T 6b521522c4784d2ab4056268823f1099 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:29.113829 1177 ts_tablet_manager.cc:1397] T 6b521522c4784d2ab4056268823f1099 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20250112 02:52:29.114655 1177 raft_consensus.cc:357] T 6b521522c4784d2ab4056268823f1099 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.114903 1177 raft_consensus.cc:383] T 6b521522c4784d2ab4056268823f1099 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.114974 1177 raft_consensus.cc:738] T 6b521522c4784d2ab4056268823f1099 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.115959 1172 ts_tablet_manager.cc:1918] T 54033a747d5a49b0a1c16f138158f596 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:29.116076 1172 log.cc:1198] T 54033a747d5a49b0a1c16f138158f596 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/54033a747d5a49b0a1c16f138158f596
I20250112 02:52:29.116492 1172 ts_tablet_manager.cc:1939] T 54033a747d5a49b0a1c16f138158f596 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:29.117362 1177 consensus_queue.cc:260] T 6b521522c4784d2ab4056268823f1099 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.117925 1177 ts_tablet_manager.cc:1428] T 6b521522c4784d2ab4056268823f1099 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:29.119695 1171 ts_tablet_manager.cc:1918] T e9fc9a4f1e8944abb76dad6c2ad53d35 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:29.119838 1171 log.cc:1198] T e9fc9a4f1e8944abb76dad6c2ad53d35 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/e9fc9a4f1e8944abb76dad6c2ad53d35
I20250112 02:52:29.120263 1171 ts_tablet_manager.cc:1939] T e9fc9a4f1e8944abb76dad6c2ad53d35 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:29.120918 1177 tablet_bootstrap.cc:492] T f6803d0f553b4204b415b2734286bf4a P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:29.123445 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet e9fc9a4f1e8944abb76dad6c2ad53d35 (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]) successfully deleted
I20250112 02:52:29.123539 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b521522c4784d2ab4056268823f1099 with delete_type TABLET_DATA_DELETED (replaced by tablet ab2e31a65294407bb62b13e92898a7d2) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:29.123782 1171 tablet_replica.cc:331] T 6b521522c4784d2ab4056268823f1099 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:29.124150 1171 raft_consensus.cc:2238] T 6b521522c4784d2ab4056268823f1099 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.124370 1171 raft_consensus.cc:2267] T 6b521522c4784d2ab4056268823f1099 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.125465 1171 ts_tablet_manager.cc:1905] T 6b521522c4784d2ab4056268823f1099 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.126544 1173 tablet_bootstrap.cc:492] T 5601a9398c9a466582a98967c0e749e2 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:29.126672 1173 ts_tablet_manager.cc:1397] T 5601a9398c9a466582a98967c0e749e2 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.004s
I20250112 02:52:29.127408 1173 raft_consensus.cc:357] T 5601a9398c9a466582a98967c0e749e2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.127627 1173 raft_consensus.cc:383] T 5601a9398c9a466582a98967c0e749e2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.127722 1173 raft_consensus.cc:738] T 5601a9398c9a466582a98967c0e749e2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.127934 1173 consensus_queue.cc:260] T 5601a9398c9a466582a98967c0e749e2 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.128412 1173 ts_tablet_manager.cc:1428] T 5601a9398c9a466582a98967c0e749e2 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:29.129405 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet b6dc79f4d1194f81963a8094ed6b0d2e (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]) successfully deleted
W20250112 02:52:29.129763 512 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b6dc79f4d1194f81963a8094ed6b0d2e because the tablet was not found. No further retry: Not found: Tablet not found: b6dc79f4d1194f81963a8094ed6b0d2e
I20250112 02:52:29.129993 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 54033a747d5a49b0a1c16f138158f596 (table default.Table_4f66980b946046a4bd2531c4ab094352 [id=c11cab752ad14d53b9b09fbaaf9b6f36]) successfully deleted
W20250112 02:52:29.130216 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet daef3a58605e40978f7389698b22265d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet daef3a58605e40978f7389698b22265d already in progress: creating tablet
I20250112 02:52:29.130741 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 9427951452104a7db765437493b21e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:29.132248 1177 tablet_bootstrap.cc:654] T f6803d0f553b4204b415b2734286bf4a P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.133265 1171 ts_tablet_manager.cc:1918] T 6b521522c4784d2ab4056268823f1099 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 02:52:29.134881 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 352c87f1dc724e7b94bc2e64abfce8d7 consensus state has no leader
I20250112 02:52:29.137022 1173 tablet_bootstrap.cc:492] T c69587c5a09d40f0baba5c2f6e8a7b41 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:29.138656 1173 tablet_bootstrap.cc:654] T c69587c5a09d40f0baba5c2f6e8a7b41 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.141572 1175 ts_tablet_manager.cc:1918] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:29.141717 1175 log.cc:1198] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/b2d8ea2ef18a46508371b4789ab6f4fa
I20250112 02:52:29.142076 1175 ts_tablet_manager.cc:1939] T b2d8ea2ef18a46508371b4789ab6f4fa P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:29.142205 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_8034217cf2a247d5af40b1123517157f" } new_table_name: "default.Table_aef01bf228ad41138e2ffe97ce0e906a" modify_external_catalogs: true
I20250112 02:52:29.142469 1172 tablet_replica.cc:331] T 9427951452104a7db765437493b21e90 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:29.142524 1171 log.cc:1198] T 6b521522c4784d2ab4056268823f1099 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/6b521522c4784d2ab4056268823f1099
I20250112 02:52:29.142652 1172 raft_consensus.cc:2238] T 9427951452104a7db765437493b21e90 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.142792 1172 raft_consensus.cc:2267] T 9427951452104a7db765437493b21e90 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.142823 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 270) took 2133 ms. Trace:
I20250112 02:52:29.142910 1171 ts_tablet_manager.cc:1939] T 6b521522c4784d2ab4056268823f1099 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:29.143503 1172 ts_tablet_manager.cc:1905] T 9427951452104a7db765437493b21e90 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.144042 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 6b521522c4784d2ab4056268823f1099 (table default.Table_3321055e22c3410eb5e5fe8be62e004c [id=f8cd4289f89a4a748c26a7a01236406e]) successfully deleted
W20250112 02:52:29.157307 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 06edfca60eed4dd3a2dee7b662b0970c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06edfca60eed4dd3a2dee7b662b0970c already in progress: creating tablet
I20250112 02:52:29.158941 1172 ts_tablet_manager.cc:1918] T 9427951452104a7db765437493b21e90 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:29.159071 1172 log.cc:1198] T 9427951452104a7db765437493b21e90 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/9427951452104a7db765437493b21e90
W20250112 02:52:29.159338 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet fe614eaeb793473d9b2da206b7345562 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe614eaeb793473d9b2da206b7345562 already in progress: creating tablet
I20250112 02:52:29.159420 1172 ts_tablet_manager.cc:1939] T 9427951452104a7db765437493b21e90 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:29.159981 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet b885247d2cb449e0a29f6f5d81518589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:29.161499 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 5601a9398c9a466582a98967c0e749e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:50844
W20250112 02:52:29.161676 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37862023095644bab2403576e8cb8c58 consensus state has no leader
I20250112 02:52:29.142905 1175 rpcz_store.cc:276] 0112 02:52:27.009562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.009625 (+ 63us) service_pool.cc:225] Handling call
0112 02:52:29.083685 (+2074060us) ts_tablet_manager.cc:1145] Deleting tablet b2d8ea2ef18a46508371b4789ab6f4fa
0112 02:52:29.083696 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:29.138318 (+ 54622us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:29.141467 (+ 3149us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:29.142770 (+ 1303us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2073808}
I20250112 02:52:29.163203 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet b2d8ea2ef18a46508371b4789ab6f4fa (table default.Table_12a25f7a620047b69f34be9eeae64dd2 [id=87c1bff49b4346599bf98166ce8672d4]) successfully deleted
I20250112 02:52:29.163614 1175 tablet_replica.cc:331] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:29.163663 1177 tablet_bootstrap.cc:492] T f6803d0f553b4204b415b2734286bf4a P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:29.163851 1177 ts_tablet_manager.cc:1397] T f6803d0f553b4204b415b2734286bf4a P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.005s
I20250112 02:52:29.163908 1175 raft_consensus.cc:2238] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.164176 1175 raft_consensus.cc:2267] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.164734 1175 ts_tablet_manager.cc:1905] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.164785 1177 raft_consensus.cc:357] T f6803d0f553b4204b415b2734286bf4a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.165043 1177 raft_consensus.cc:383] T f6803d0f553b4204b415b2734286bf4a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.165146 1177 raft_consensus.cc:738] T f6803d0f553b4204b415b2734286bf4a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.165421 1177 consensus_queue.cc:260] T f6803d0f553b4204b415b2734286bf4a P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.166277 1177 ts_tablet_manager.cc:1428] T f6803d0f553b4204b415b2734286bf4a P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:29.166654 1177 tablet_bootstrap.cc:492] T a1a0a13696fc41ccb28218f1275a4b5d P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:29.168336 1177 tablet_bootstrap.cc:654] T a1a0a13696fc41ccb28218f1275a4b5d P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.168596 1172 tablet_replica.cc:331] T b885247d2cb449e0a29f6f5d81518589 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:29.168812 1172 raft_consensus.cc:2238] T b885247d2cb449e0a29f6f5d81518589 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.169688 1172 raft_consensus.cc:2267] T b885247d2cb449e0a29f6f5d81518589 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.172950 754 tablet_service.cc:1514] Processing DeleteTablet for tablet f6803d0f553b4204b415b2734286bf4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:29.173612 1171 tablet_replica.cc:331] T f6803d0f553b4204b415b2734286bf4a P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:29.173861 1171 raft_consensus.cc:2238] T f6803d0f553b4204b415b2734286bf4a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.174124 1171 raft_consensus.cc:2267] T f6803d0f553b4204b415b2734286bf4a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.174841 1171 ts_tablet_manager.cc:1905] T f6803d0f553b4204b415b2734286bf4a P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.175403 1177 tablet_bootstrap.cc:492] T a1a0a13696fc41ccb28218f1275a4b5d P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:29.175679 1177 ts_tablet_manager.cc:1397] T a1a0a13696fc41ccb28218f1275a4b5d P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20250112 02:52:29.176702 1177 raft_consensus.cc:357] T a1a0a13696fc41ccb28218f1275a4b5d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.177105 1177 raft_consensus.cc:383] T a1a0a13696fc41ccb28218f1275a4b5d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.177213 1177 raft_consensus.cc:738] T a1a0a13696fc41ccb28218f1275a4b5d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.177493 1177 consensus_queue.cc:260] T a1a0a13696fc41ccb28218f1275a4b5d P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.178092 1177 ts_tablet_manager.cc:1428] T a1a0a13696fc41ccb28218f1275a4b5d P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:29.178428 1177 tablet_bootstrap.cc:492] T afa2f8aabe5e42ccbc15b193fdc03563 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:29.180145 1177 tablet_bootstrap.cc:654] T afa2f8aabe5e42ccbc15b193fdc03563 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.181382 1172 ts_tablet_manager.cc:1905] T b885247d2cb449e0a29f6f5d81518589 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.182381 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 9427951452104a7db765437493b21e90 (table default.Table_d38ef91002ba4653916c476d1df3a9c6 [id=fd0dc6c5d5d14251a3e6b52e3d66ed0e]) successfully deleted
I20250112 02:52:29.187343 1172 ts_tablet_manager.cc:1918] T b885247d2cb449e0a29f6f5d81518589 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:29.180876 754 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a0a13696fc41ccb28218f1275a4b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:29.190831 1172 log.cc:1198] T b885247d2cb449e0a29f6f5d81518589 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/b885247d2cb449e0a29f6f5d81518589
I20250112 02:52:29.191236 1172 ts_tablet_manager.cc:1939] T b885247d2cb449e0a29f6f5d81518589 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:29.191643 1172 tablet_replica.cc:331] T 5601a9398c9a466582a98967c0e749e2 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:29.191812 1172 raft_consensus.cc:2238] T 5601a9398c9a466582a98967c0e749e2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.191951 1172 raft_consensus.cc:2267] T 5601a9398c9a466582a98967c0e749e2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.200060 1181 tablet_bootstrap.cc:492] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:29.200230 1181 ts_tablet_manager.cc:1397] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.166s user 0.002s sys 0.000s
I20250112 02:52:29.200738 1172 ts_tablet_manager.cc:1905] T 5601a9398c9a466582a98967c0e749e2 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.200816 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_849b7f79a5784f5fabed0136deb0a129"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:29.201027 1181 raft_consensus.cc:357] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.201243 1181 raft_consensus.cc:383] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.201331 1181 raft_consensus.cc:738] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.201573 1181 consensus_queue.cc:260] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:29.201702 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_849b7f79a5784f5fabed0136deb0a129 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:29.202101 1181 ts_tablet_manager.cc:1428] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:29.202437 1181 tablet_bootstrap.cc:492] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:29.203716 1173 tablet_bootstrap.cc:492] T c69587c5a09d40f0baba5c2f6e8a7b41 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:29.203869 1173 ts_tablet_manager.cc:1397] T c69587c5a09d40f0baba5c2f6e8a7b41 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.067s user 0.021s sys 0.009s
I20250112 02:52:29.204905 1177 tablet_bootstrap.cc:492] T afa2f8aabe5e42ccbc15b193fdc03563 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:29.205034 1177 ts_tablet_manager.cc:1397] T afa2f8aabe5e42ccbc15b193fdc03563 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20250112 02:52:29.205062 1173 raft_consensus.cc:357] T c69587c5a09d40f0baba5c2f6e8a7b41 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.205267 1173 raft_consensus.cc:383] T c69587c5a09d40f0baba5c2f6e8a7b41 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.205328 1173 raft_consensus.cc:738] T c69587c5a09d40f0baba5c2f6e8a7b41 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.205751 1177 raft_consensus.cc:357] T afa2f8aabe5e42ccbc15b193fdc03563 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.205776 1173 consensus_queue.cc:260] T c69587c5a09d40f0baba5c2f6e8a7b41 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.205933 1177 raft_consensus.cc:383] T afa2f8aabe5e42ccbc15b193fdc03563 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.206018 1177 raft_consensus.cc:738] T afa2f8aabe5e42ccbc15b193fdc03563 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.206245 1173 ts_tablet_manager.cc:1428] T c69587c5a09d40f0baba5c2f6e8a7b41 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:29.206223 1177 consensus_queue.cc:260] T afa2f8aabe5e42ccbc15b193fdc03563 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.206532 1173 tablet_bootstrap.cc:492] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:29.206848 1177 ts_tablet_manager.cc:1428] T afa2f8aabe5e42ccbc15b193fdc03563 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:29.207079 1177 tablet_bootstrap.cc:492] T 1726d61cc36a4deb83c5d604e8daf4f0 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:29.208105 1173 tablet_bootstrap.cc:654] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.208669 1177 tablet_bootstrap.cc:654] T 1726d61cc36a4deb83c5d604e8daf4f0 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.210460 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_f7fd6e154c104f7ab66a01e7f8202da6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:29.210950 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7fd6e154c104f7ab66a01e7f8202da6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:29.212033 526 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_d0a145ff8f4d4e0484e015fb4d18860c" } modify_external_catalogs: true
I20250112 02:52:29.212225 526 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_d0a145ff8f4d4e0484e015fb4d18860c" } modify_external_catalogs: true
I20250112 02:52:29.214560 1172 ts_tablet_manager.cc:1918] T 5601a9398c9a466582a98967c0e749e2 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:29.214684 1172 log.cc:1198] T 5601a9398c9a466582a98967c0e749e2 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/5601a9398c9a466582a98967c0e749e2
I20250112 02:52:29.214792 893 tablet_service.cc:1514] Processing DeleteTablet for tablet b9199b0d2b924a79938a979df4196bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:29.215037 1172 ts_tablet_manager.cc:1939] T 5601a9398c9a466582a98967c0e749e2 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:29.219719 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_692d9e1bfcfd435e8c420696c1e5aae5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:29.220202 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_692d9e1bfcfd435e8c420696c1e5aae5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:29.222667 754 tablet_service.cc:1514] Processing DeleteTablet for tablet afa2f8aabe5e42ccbc15b193fdc03563 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:29.224309 1173 tablet_bootstrap.cc:492] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:29.224444 1173 ts_tablet_manager.cc:1397] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20250112 02:52:29.225299 1173 raft_consensus.cc:357] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.225553 1173 raft_consensus.cc:383] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.226169 1173 raft_consensus.cc:738] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.226511 1173 consensus_queue.cc:260] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.227124 1173 ts_tablet_manager.cc:1428] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:29.228217 1177 tablet_bootstrap.cc:492] T 1726d61cc36a4deb83c5d604e8daf4f0 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:29.228431 1177 ts_tablet_manager.cc:1397] T 1726d61cc36a4deb83c5d604e8daf4f0 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250112 02:52:29.229183 1177 raft_consensus.cc:357] T 1726d61cc36a4deb83c5d604e8daf4f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.229430 1177 raft_consensus.cc:383] T 1726d61cc36a4deb83c5d604e8daf4f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.229532 1177 raft_consensus.cc:738] T 1726d61cc36a4deb83c5d604e8daf4f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.229723 1173 tablet_bootstrap.cc:492] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:29.229857 1177 consensus_queue.cc:260] T 1726d61cc36a4deb83c5d604e8daf4f0 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.230367 1177 ts_tablet_manager.cc:1428] T 1726d61cc36a4deb83c5d604e8daf4f0 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:29.230702 1177 tablet_bootstrap.cc:492] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:29.231722 1173 tablet_bootstrap.cc:654] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.232378 1177 tablet_bootstrap.cc:654] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.237425 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_ec58001618964e3bacdaeaed93191082" } new_table_name: "default.Table_4001552c6dc246e18a8b0ad9041267c6" modify_external_catalogs: true
I20250112 02:52:29.244349 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 1726d61cc36a4deb83c5d604e8daf4f0 with delete_type TABLET_DATA_DELETED (replaced by tablet b2ccb88108f24e36ba4b7ebf36398950) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:29.257537 1537 raft_consensus.cc:491] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:29.257687 1537 raft_consensus.cc:513] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.258255 1537 leader_election.cc:290] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:29.258782 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b521522c4784d2ab4056268823f1099" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:29.258996 918 raft_consensus.cc:2463] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:29.259400 989 leader_election.cc:304] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:29.259732 1537 raft_consensus.cc:2798] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:29.259832 1537 raft_consensus.cc:491] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:29.259946 1537 raft_consensus.cc:3054] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:29.262774 1537 raft_consensus.cc:513] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:29.263156 991 leader_election.cc:343] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: 6b521522c4784d2ab4056268823f1099
I20250112 02:52:29.264005 519 master_service.cc:946] ReplaceTablet: received request to replace tablet 1c639a2bcd4740e4be7b2b162004669a from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:29.264040 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b521522c4784d2ab4056268823f1099" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:29.264189 918 raft_consensus.cc:3054] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:29.264536 1537 leader_election.cc:290] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:29.265811 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_06b4717d1d6d4b73b1488df2da81bbdc" } new_table_name: "default.Table_829049fe22b74e62a87dd1bed65fb3fe" modify_external_catalogs: true
I20250112 02:52:29.266656 918 raft_consensus.cc:2463] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:29.267105 989 leader_election.cc:304] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
W20250112 02:52:29.267596 991 leader_election.cc:343] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: 6b521522c4784d2ab4056268823f1099
I20250112 02:52:29.268704 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_4c59f5d61aab4b6184f0d9a4e21f965f" } modify_external_catalogs: true
I20250112 02:52:29.268865 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_4c59f5d61aab4b6184f0d9a4e21f965f" } modify_external_catalogs: true
I20250112 02:52:29.270969 1537 raft_consensus.cc:2798] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:29.276659 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_0b48267c6ffc44989b5335e25e559c0d" } new_table_name: "default.Table_bfb5e14e3f334af2a4459f8f3cef66a8" modify_external_catalogs: true
I20250112 02:52:29.282894 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet b885247d2cb449e0a29f6f5d81518589 (table default.Table_edaa8b50dfa8463e99e05ab8d1d89bc3 [id=a6a18be9a78e4959b7d272c447f6fc25]) successfully deleted
I20250112 02:52:29.283545 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 5601a9398c9a466582a98967c0e749e2 (table default.Table_3b1c87a5169e49b68fdf3d5c662a27bc [id=ed3672fa460a4aa2ae1d3b643ade43e6]) successfully deleted
I20250112 02:52:29.291255 893 tablet_service.cc:1467] Processing CreateTablet for tablet b124c15736084e6d8f69b4fb7c41cf39 (DEFAULT_TABLE table=default.Table_f0fb142b9aab490bb067d3cf64dc47b1 [id=1991c0821ef4478eb0eb9282ab772c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.291711 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b124c15736084e6d8f69b4fb7c41cf39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.293746 759 tablet_service.cc:1467] Processing CreateTablet for tablet 9ded8bcdda914ea6bc4f9bb92d413dee (DEFAULT_TABLE table=default.Table_b0d877b624604ddc93c56bec6e05261f [id=510ea5b8b1ae461faaa7c9787162f29a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.293859 754 tablet_service.cc:1467] Processing CreateTablet for tablet b124c15736084e6d8f69b4fb7c41cf39 (DEFAULT_TABLE table=default.Table_f0fb142b9aab490bb067d3cf64dc47b1 [id=1991c0821ef4478eb0eb9282ab772c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.294082 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ded8bcdda914ea6bc4f9bb92d413dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.294836 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b124c15736084e6d8f69b4fb7c41cf39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.296005 899 tablet_service.cc:1467] Processing CreateTablet for tablet 9ded8bcdda914ea6bc4f9bb92d413dee (DEFAULT_TABLE table=default.Table_b0d877b624604ddc93c56bec6e05261f [id=510ea5b8b1ae461faaa7c9787162f29a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.296322 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ded8bcdda914ea6bc4f9bb92d413dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.299780 753 tablet_service.cc:1467] Processing CreateTablet for tablet bd5b3c712c0f448ea9b14ab93dd731ab (DEFAULT_TABLE table=default.Table_9c10574f7c7d435890eb9a24af23bcb9 [id=c0e16b77525146a28b8182137bc81566]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.300155 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd5b3c712c0f448ea9b14ab93dd731ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.301573 889 tablet_service.cc:1467] Processing CreateTablet for tablet bd5b3c712c0f448ea9b14ab93dd731ab (DEFAULT_TABLE table=default.Table_9c10574f7c7d435890eb9a24af23bcb9 [id=c0e16b77525146a28b8182137bc81566]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.301918 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd5b3c712c0f448ea9b14ab93dd731ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.302353 752 tablet_service.cc:1467] Processing CreateTablet for tablet 3d662e0ce13f447595e5f92b94de3018 (DEFAULT_TABLE table=default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.302685 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d662e0ce13f447595e5f92b94de3018. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.303691 885 tablet_service.cc:1467] Processing CreateTablet for tablet 8429a3d8636d4e2ca4c1191f85b9266b (DEFAULT_TABLE table=default.Table_328664e3c01f483ca8e5ff50a497f68e [id=1f5c5f9f5e664250898a3935787b6b51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.303848 757 tablet_service.cc:1467] Processing CreateTablet for tablet 8429a3d8636d4e2ca4c1191f85b9266b (DEFAULT_TABLE table=default.Table_328664e3c01f483ca8e5ff50a497f68e [id=1f5c5f9f5e664250898a3935787b6b51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.304005 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8429a3d8636d4e2ca4c1191f85b9266b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.304267 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8429a3d8636d4e2ca4c1191f85b9266b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.306886 756 tablet_service.cc:1467] Processing CreateTablet for tablet cec5346e13164274ae7bc3b6aa241f61 (DEFAULT_TABLE table=default.Table_4965c2235c4e422889169067ff632d29 [id=8dd987fcd63c4ee7a5fcabd32affaf3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.307210 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cec5346e13164274ae7bc3b6aa241f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.307783 751 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6650515ea540a8bbf21e2f88363360 (DEFAULT_TABLE table=default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.308192 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a6650515ea540a8bbf21e2f88363360. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.308751 747 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1954e7859f422f9690fe4a9384a8bb (DEFAULT_TABLE table=default.Table_06b4717d1d6d4b73b1488df2da81bbdc [id=2e892eba68234ffca0be4ef3e49cf309]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.309238 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e1954e7859f422f9690fe4a9384a8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.310695 898 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6650515ea540a8bbf21e2f88363360 (DEFAULT_TABLE table=default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.310999 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a6650515ea540a8bbf21e2f88363360. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.311728 755 tablet_service.cc:1467] Processing CreateTablet for tablet 7f3be6bcbe8a45f0a8076b4842d90dc2 (DEFAULT_TABLE table=default.Table_6d12502c6e714726b9d807690fd21ea4 [id=9b5607155d264cf2879d18305398df11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.312117 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f3be6bcbe8a45f0a8076b4842d90dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.315677 896 tablet_service.cc:1467] Processing CreateTablet for tablet 7f3be6bcbe8a45f0a8076b4842d90dc2 (DEFAULT_TABLE table=default.Table_6d12502c6e714726b9d807690fd21ea4 [id=9b5607155d264cf2879d18305398df11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.315966 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f3be6bcbe8a45f0a8076b4842d90dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.319746 750 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad9c47fcb534619949971796b60eba2 (DEFAULT_TABLE table=default.Table_569c0e413a014e52bcb98a9f4f644479 [id=75c94dcaccf344abbac5808bab28a673]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.320070 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ad9c47fcb534619949971796b60eba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.323753 758 tablet_service.cc:1467] Processing CreateTablet for tablet 26577b614e32458e8bc5d6830c8c2e40 (DEFAULT_TABLE table=default.Table_03e3448d809c4fc792d0bff94a713044 [id=5f3cc73b47414c438591107e35a12472]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.324082 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26577b614e32458e8bc5d6830c8c2e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.327026 890 tablet_service.cc:1467] Processing CreateTablet for tablet 26577b614e32458e8bc5d6830c8c2e40 (DEFAULT_TABLE table=default.Table_03e3448d809c4fc792d0bff94a713044 [id=5f3cc73b47414c438591107e35a12472]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.327332 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26577b614e32458e8bc5d6830c8c2e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.331779 749 tablet_service.cc:1467] Processing CreateTablet for tablet 03a32b424ec4431d88f6af45dd09089f (DEFAULT_TABLE table=default.Table_f173732a21194c7aa8f9a40eb7eb07a1 [id=e04012ba08a640828b73c52803bf214b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.332113 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03a32b424ec4431d88f6af45dd09089f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.333496 759 tablet_service.cc:1467] Processing CreateTablet for tablet 82b351c563d749df806ab5d0331a4ef2 (DEFAULT_TABLE table=default.Table_4603068c01544e2e965dca5177e36b40 [id=8502f21e9b61436093db1241cfbb5ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.333818 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82b351c563d749df806ab5d0331a4ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.335412 894 tablet_service.cc:1467] Processing CreateTablet for tablet 03a32b424ec4431d88f6af45dd09089f (DEFAULT_TABLE table=default.Table_f173732a21194c7aa8f9a40eb7eb07a1 [id=e04012ba08a640828b73c52803bf214b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.335752 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03a32b424ec4431d88f6af45dd09089f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.339015 760 tablet_service.cc:1467] Processing CreateTablet for tablet 41d5799b5b36487da42dc43b1c70ae26 (DEFAULT_TABLE table=default.Table_2b5aeec3b0bf40aea08010f65e422c2c [id=e292141365e9453fa1816e0f313b2300]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.339082 745 tablet_service.cc:1467] Processing CreateTablet for tablet 61eac6b572ff47c185768bee7490edf7 (DEFAULT_TABLE table=default.Table_e8e62bd2fffe4f50849f722dbece2a2c [id=3603f999c77a48dca737bc8ff74eb9de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.339321 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41d5799b5b36487da42dc43b1c70ae26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.339540 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61eac6b572ff47c185768bee7490edf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.340003 748 tablet_service.cc:1467] Processing CreateTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff (DEFAULT_TABLE table=default.Table_d0a145ff8f4d4e0484e015fb4d18860c [id=aba940f2fbb849168f3cb1bfd1c36e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.323414 886 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad9c47fcb534619949971796b60eba2 (DEFAULT_TABLE table=default.Table_569c0e413a014e52bcb98a9f4f644479 [id=75c94dcaccf344abbac5808bab28a673]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.341202 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ad9c47fcb534619949971796b60eba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.342387 523 master_service.cc:946] ReplaceTablet: received request to replace tablet b6d84027ad6d40a5be5afcdd854431a5 from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:29.343652 526 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet f776e2a6ec704cf9b64483bc7d5b1dff
I20250112 02:52:29.343786 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet d659c5a4f7a446c6aaf2283f65b45106
I20250112 02:52:29.344296 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_83212f303f0b4413aa6c62887da9dd34" } modify_external_catalogs: true
I20250112 02:52:29.344367 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_6b8cb70a5e5b4519af5d247f6b5bb8b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:29.344473 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_83212f303f0b4413aa6c62887da9dd34" } modify_external_catalogs: true
W20250112 02:52:29.344849 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b8cb70a5e5b4519af5d247f6b5bb8b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:29.345805 746 tablet_service.cc:1467] Processing CreateTablet for tablet ea18666ec324403b9f10a099688438b8 (DEFAULT_TABLE table=default.Table_e1b086de31974b33b7ac824ebeb757ee [id=36b5d7e7c4444808921063d46a50da46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.346954 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet d45fce86e02f46d4a57af718fa2a62c8
I20250112 02:52:29.347257 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f776e2a6ec704cf9b64483bc7d5b1dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.347465 522 catalog_manager.cc:6596] ReplaceTablet: tablet d45fce86e02f46d4a57af718fa2a62c8 deleted and replaced by tablet 5804fc4130f84c69832c1022b7bd7fa5
I20250112 02:52:29.347783 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_3a84890c771c4574a25d33543fe7217d" } modify_external_catalogs: true
W20250112 02:52:29.347875 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5804fc4130f84c69832c1022b7bd7fa5 has no consensus state
I20250112 02:52:29.347949 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_3a84890c771c4574a25d33543fe7217d" } modify_external_catalogs: true
I20250112 02:52:29.348277 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 96e201d798c94e448a78e1c8adf41724
W20250112 02:52:29.350574 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e1954e7859f422f9690fe4a9384a8bb consensus state has no leader
I20250112 02:52:29.351557 521 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c69587c5a09d40f0baba5c2f6e8a7b41 (table default.Table_1216acc1fa6b4e51a55a89ffe22bcc9a [id=b5457e4343ee43618c8309e737b69ec3]). Expected version 1 got 0
I20250112 02:52:29.352219 895 tablet_service.cc:1467] Processing CreateTablet for tablet 41d5799b5b36487da42dc43b1c70ae26 (DEFAULT_TABLE table=default.Table_2b5aeec3b0bf40aea08010f65e422c2c [id=e292141365e9453fa1816e0f313b2300]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.352555 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41d5799b5b36487da42dc43b1c70ae26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.352483 897 tablet_service.cc:1467] Processing CreateTablet for tablet cec5346e13164274ae7bc3b6aa241f61 (DEFAULT_TABLE table=default.Table_4965c2235c4e422889169067ff632d29 [id=8dd987fcd63c4ee7a5fcabd32affaf3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.353032 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cec5346e13164274ae7bc3b6aa241f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.355829 887 tablet_service.cc:1467] Processing CreateTablet for tablet 3d662e0ce13f447595e5f92b94de3018 (DEFAULT_TABLE table=default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.356148 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d662e0ce13f447595e5f92b94de3018. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.358300 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 96e201d798c94e448a78e1c8adf41724 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:29.358557 881 tablet_service.cc:1514] Processing DeleteTablet for tablet d45fce86e02f46d4a57af718fa2a62c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 5804fc4130f84c69832c1022b7bd7fa5) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:29.358810 882 tablet_service.cc:1514] Processing DeleteTablet for tablet d659c5a4f7a446c6aaf2283f65b45106 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:29.359076 883 tablet_service.cc:1514] Processing DeleteTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:29.359541 884 tablet_service.cc:1467] Processing CreateTablet for tablet 82b351c563d749df806ab5d0331a4ef2 (DEFAULT_TABLE table=default.Table_4603068c01544e2e965dca5177e36b40 [id=8502f21e9b61436093db1241cfbb5ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.360510 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82b351c563d749df806ab5d0331a4ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.367592 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_63abe3ac694a4ed6bf4aef720b542e73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:29.368060 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63abe3ac694a4ed6bf4aef720b542e73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:29.369275 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_c11c44af47604f9589b15d5c6a97c255"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:29.369751 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c11c44af47604f9589b15d5c6a97c255 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:29.371399 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_9284616ca0cd499cb51df0322db9948f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:29.371696 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_a56064e2c6b449b7b1e32f2a69f2f7fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:29.372098 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a56064e2c6b449b7b1e32f2a69f2f7fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:29.375298 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_5d1f4404d0cc46268126e703afb52565"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:29.375571 744 tablet_service.cc:1514] Processing DeleteTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:29.375741 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d1f4404d0cc46268126e703afb52565 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:29.376473 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9284616ca0cd499cb51df0322db9948f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:29.376792 1033 tablet_service.cc:1467] Processing CreateTablet for tablet b124c15736084e6d8f69b4fb7c41cf39 (DEFAULT_TABLE table=default.Table_f0fb142b9aab490bb067d3cf64dc47b1 [id=1991c0821ef4478eb0eb9282ab772c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.377198 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b124c15736084e6d8f69b4fb7c41cf39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.377496 1173 tablet_bootstrap.cc:492] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:29.377641 1173 ts_tablet_manager.cc:1397] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.148s user 0.025s sys 0.031s
I20250112 02:52:29.378376 1173 raft_consensus.cc:357] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.378636 1173 raft_consensus.cc:383] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.378731 1173 raft_consensus.cc:738] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.378988 1173 consensus_queue.cc:260] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.379987 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 9ded8bcdda914ea6bc4f9bb92d413dee (DEFAULT_TABLE table=default.Table_b0d877b624604ddc93c56bec6e05261f [id=510ea5b8b1ae461faaa7c9787162f29a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.380365 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ded8bcdda914ea6bc4f9bb92d413dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.380328 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6650515ea540a8bbf21e2f88363360 (DEFAULT_TABLE table=default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.380661 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a6650515ea540a8bbf21e2f88363360. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.380870 1019 tablet_service.cc:1467] Processing CreateTablet for tablet bd5b3c712c0f448ea9b14ab93dd731ab (DEFAULT_TABLE table=default.Table_9c10574f7c7d435890eb9a24af23bcb9 [id=c0e16b77525146a28b8182137bc81566]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.381233 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd5b3c712c0f448ea9b14ab93dd731ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.383742 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 8429a3d8636d4e2ca4c1191f85b9266b (DEFAULT_TABLE table=default.Table_328664e3c01f483ca8e5ff50a497f68e [id=1f5c5f9f5e664250898a3935787b6b51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.384071 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8429a3d8636d4e2ca4c1191f85b9266b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.386031 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 61eac6b572ff47c185768bee7490edf7 (DEFAULT_TABLE table=default.Table_e8e62bd2fffe4f50849f722dbece2a2c [id=3603f999c77a48dca737bc8ff74eb9de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.386394 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61eac6b572ff47c185768bee7490edf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.387037 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 7f3be6bcbe8a45f0a8076b4842d90dc2 (DEFAULT_TABLE table=default.Table_6d12502c6e714726b9d807690fd21ea4 [id=9b5607155d264cf2879d18305398df11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.387434 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f3be6bcbe8a45f0a8076b4842d90dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.387710 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad9c47fcb534619949971796b60eba2 (DEFAULT_TABLE table=default.Table_569c0e413a014e52bcb98a9f4f644479 [id=75c94dcaccf344abbac5808bab28a673]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.388211 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 26577b614e32458e8bc5d6830c8c2e40 (DEFAULT_TABLE table=default.Table_03e3448d809c4fc792d0bff94a713044 [id=5f3cc73b47414c438591107e35a12472]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.388724 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 03a32b424ec4431d88f6af45dd09089f (DEFAULT_TABLE table=default.Table_f173732a21194c7aa8f9a40eb7eb07a1 [id=e04012ba08a640828b73c52803bf214b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.389211 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 41d5799b5b36487da42dc43b1c70ae26 (DEFAULT_TABLE table=default.Table_2b5aeec3b0bf40aea08010f65e422c2c [id=e292141365e9453fa1816e0f313b2300]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.389674 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 3d662e0ce13f447595e5f92b94de3018 (DEFAULT_TABLE table=default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.390158 1034 tablet_service.cc:1467] Processing CreateTablet for tablet cec5346e13164274ae7bc3b6aa241f61 (DEFAULT_TABLE table=default.Table_4965c2235c4e422889169067ff632d29 [id=8dd987fcd63c4ee7a5fcabd32affaf3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.390630 1020 tablet_service.cc:1467] Processing CreateTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff (DEFAULT_TABLE table=default.Table_d0a145ff8f4d4e0484e015fb4d18860c [id=aba940f2fbb849168f3cb1bfd1c36e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.391122 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1954e7859f422f9690fe4a9384a8bb (DEFAULT_TABLE table=default.Table_06b4717d1d6d4b73b1488df2da81bbdc [id=2e892eba68234ffca0be4ef3e49cf309]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.391713 1022 tablet_service.cc:1467] Processing CreateTablet for tablet ea18666ec324403b9f10a099688438b8 (DEFAULT_TABLE table=default.Table_e1b086de31974b33b7ac824ebeb757ee [id=36b5d7e7c4444808921063d46a50da46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.392235 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 82b351c563d749df806ab5d0331a4ef2 (DEFAULT_TABLE table=default.Table_4603068c01544e2e965dca5177e36b40 [id=8502f21e9b61436093db1241cfbb5ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.392591 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet d659c5a4f7a446c6aaf2283f65b45106 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:29.392827 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:29.393005 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet d45fce86e02f46d4a57af718fa2a62c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 5804fc4130f84c69832c1022b7bd7fa5) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:29.393239 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 96e201d798c94e448a78e1c8adf41724 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:29.395278 1172 tablet_replica.cc:331] T 96e201d798c94e448a78e1c8adf41724 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:29.395512 1173 ts_tablet_manager.cc:1428] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.018s user 0.003s sys 0.004s
I20250112 02:52:29.395612 892 tablet_service.cc:1467] Processing CreateTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff (DEFAULT_TABLE table=default.Table_d0a145ff8f4d4e0484e015fb4d18860c [id=aba940f2fbb849168f3cb1bfd1c36e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.395792 1173 tablet_bootstrap.cc:492] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:29.395972 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f776e2a6ec704cf9b64483bc7d5b1dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.397294 897 tablet_service.cc:1467] Processing CreateTablet for tablet 61eac6b572ff47c185768bee7490edf7 (DEFAULT_TABLE table=default.Table_e8e62bd2fffe4f50849f722dbece2a2c [id=3603f999c77a48dca737bc8ff74eb9de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.397464 1173 tablet_bootstrap.cc:654] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.397635 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61eac6b572ff47c185768bee7490edf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.398411 888 tablet_service.cc:1467] Processing CreateTablet for tablet ea18666ec324403b9f10a099688438b8 (DEFAULT_TABLE table=default.Table_e1b086de31974b33b7ac824ebeb757ee [id=36b5d7e7c4444808921063d46a50da46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.399098 891 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1954e7859f422f9690fe4a9384a8bb (DEFAULT_TABLE table=default.Table_06b4717d1d6d4b73b1488df2da81bbdc [id=2e892eba68234ffca0be4ef3e49cf309]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.399231 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea18666ec324403b9f10a099688438b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.399585 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f776e2a6ec704cf9b64483bc7d5b1dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.401160 1457 raft_consensus.cc:491] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:29.401719 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ad9c47fcb534619949971796b60eba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.401821 1457 raft_consensus.cc:513] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.402781 1458 raft_consensus.cc:491] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:29.402940 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e1954e7859f422f9690fe4a9384a8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.402909 1458 raft_consensus.cc:513] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:29.404281 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cf54f5185a0f4c70b9057309fca14e56 consensus state has no leader
I20250112 02:52:29.404642 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_ed757d89486c45058437db3268063639" } new_table_name: "default.Table_daa4ea5584bd498fafec7fff852d084c" modify_external_catalogs: true
I20250112 02:52:29.408982 744 tablet_service.cc:1514] Processing DeleteTablet for tablet d659c5a4f7a446c6aaf2283f65b45106 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:29.427517 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet b6d84027ad6d40a5be5afcdd854431a5
I20250112 02:52:29.427937 523 catalog_manager.cc:6596] ReplaceTablet: tablet b6d84027ad6d40a5be5afcdd854431a5 deleted and replaced by tablet fbe9e9624ede40c29ea852b2b5494965
I20250112 02:52:29.428346 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_73e52096e12e47bab61cbfd2c72c70b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:29.428826 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73e52096e12e47bab61cbfd2c72c70b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:29.430119 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_2db060823dc54c289c2595ad2684d694" } new_table_name: "default.Table_feaebd2f4cbb47b6b24ff79614994d6f" modify_external_catalogs: true
I20250112 02:52:29.431973 883 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d84027ad6d40a5be5afcdd854431a5 with delete_type TABLET_DATA_DELETED (replaced by tablet fbe9e9624ede40c29ea852b2b5494965) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:29.433012 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 352c87f1dc724e7b94bc2e64abfce8d7
I20250112 02:52:29.438058 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Sending DeleteTablet for 3 replicas of tablet 02d0f67141dc46b6a87ca0ad080d5bfe
W20250112 02:52:29.438696 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d25815a5df94e099e7481c48c3da793 consensus state has no leader
I20250112 02:52:29.439062 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d0f67141dc46b6a87ca0ad080d5bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:29.443315 890 tablet_service.cc:1467] Processing CreateTablet for tablet d63d411f906b4d4eb68343bdcc493e05 (DEFAULT_TABLE table=default.Table_e80769df10444dd5ae4bc23a8c6cc09e [id=d91a9f9e1f76482e99d4f10c0568f2e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.444929 1458 leader_election.cc:290] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:29.445462 1457 leader_election.cc:290] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:29.450040 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 352c87f1dc724e7b94bc2e64abfce8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:51456
I20250112 02:52:29.450663 898 tablet_service.cc:1467] Processing CreateTablet for tablet 5a3af0ea4e954053ae89814f3e74e6b7 (DEFAULT_TABLE table=default.Table_525750dad2b7461586d070dbaa74d0b1 [id=7d048aa907a74605964b68726ac5d759]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.451316 522 master_service.cc:946] ReplaceTablet: received request to replace tablet 1dda52f012794493a102ed1abc718f5b from {username='slave'} at 127.0.0.1:40934
I20250112 02:52:29.455207 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d63d411f906b4d4eb68343bdcc493e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.465056 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a3af0ea4e954053ae89814f3e74e6b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.465703 1181 tablet_bootstrap.cc:654] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.470754 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea18666ec324403b9f10a099688438b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.473856 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadb2ab1d2024f889841ac91e065e064" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:29.474328 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f22a7b1b804661af3b7e785a833739" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:29.476595 524 master_service.cc:946] ReplaceTablet: received request to replace tablet aadb2ab1d2024f889841ac91e065e064 from {username='slave'} at 127.0.0.1:40934
W20250112 02:52:29.476998 855 leader_election.cc:343] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:29.478149 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 96e201d798c94e448a78e1c8adf41724 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:29.477481 744 tablet_service.cc:1514] Processing DeleteTablet for tablet d45fce86e02f46d4a57af718fa2a62c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 5804fc4130f84c69832c1022b7bd7fa5) from {username='slave'} at 127.0.0.1:40386
W20250112 02:52:29.480533 855 leader_election.cc:343] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:29.481005 744 tablet_service.cc:1467] Processing CreateTablet for tablet 5a3af0ea4e954053ae89814f3e74e6b7 (DEFAULT_TABLE table=default.Table_525750dad2b7461586d070dbaa74d0b1 [id=7d048aa907a74605964b68726ac5d759]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.481639 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a3af0ea4e954053ae89814f3e74e6b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.482722 743 tablet_service.cc:1467] Processing CreateTablet for tablet d63d411f906b4d4eb68343bdcc493e05 (DEFAULT_TABLE table=default.Table_e80769df10444dd5ae4bc23a8c6cc09e [id=d91a9f9e1f76482e99d4f10c0568f2e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.483619 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d63d411f906b4d4eb68343bdcc493e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.484772 742 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d84027ad6d40a5be5afcdd854431a5 with delete_type TABLET_DATA_DELETED (replaced by tablet fbe9e9624ede40c29ea852b2b5494965) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:29.485394 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 352c87f1dc724e7b94bc2e64abfce8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:29.486704 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d0f67141dc46b6a87ca0ad080d5bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:29.498863 1177 tablet_bootstrap.cc:492] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:29.499008 1177 ts_tablet_manager.cc:1397] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.268s user 0.036s sys 0.074s
I20250112 02:52:29.499752 1177 raft_consensus.cc:357] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.499929 1177 raft_consensus.cc:383] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.500016 1177 raft_consensus.cc:738] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.500252 1177 consensus_queue.cc:260] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.500720 1177 ts_tablet_manager.cc:1428] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:29.505887 1177 tablet_bootstrap.cc:492] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:29.508145 1177 tablet_bootstrap.cc:654] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:29.535903 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet d659c5a4f7a446c6aaf2283f65b45106 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d659c5a4f7a446c6aaf2283f65b45106 already in progress: creating tablet
W20250112 02:52:29.536159 512 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f776e2a6ec704cf9b64483bc7d5b1dff because the tablet was not found. No further retry: Not found: Tablet not found: f776e2a6ec704cf9b64483bc7d5b1dff
W20250112 02:52:29.536372 512 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet d45fce86e02f46d4a57af718fa2a62c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d45fce86e02f46d4a57af718fa2a62c8 already in progress: creating tablet
I20250112 02:52:29.534680 1175 ts_tablet_manager.cc:1918] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:29.540014 1175 log.cc:1198] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/cb724feccc7f4fbc846fb6fa61be3385
W20250112 02:52:29.544174 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 896908b2726c40f9aabdfc9a6b3283f7 has no consensus state
I20250112 02:52:29.550837 1503 consensus_queue.cc:1035] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:29.553562 1172 raft_consensus.cc:2238] T 96e201d798c94e448a78e1c8adf41724 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.553264 1175 ts_tablet_manager.cc:1939] T cb724feccc7f4fbc846fb6fa61be3385 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:29.554690 1172 raft_consensus.cc:2267] T 96e201d798c94e448a78e1c8adf41724 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.555251 1172 ts_tablet_manager.cc:1905] T 96e201d798c94e448a78e1c8adf41724 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.555851 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d662e0ce13f447595e5f92b94de3018. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.558320 1457 raft_consensus.cc:491] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:29.558454 1457 raft_consensus.cc:513] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.559141 1457 leader_election.cc:290] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:29.559594 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c686fdc8896463c81c9f7c5f53a3da2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:29.560091 855 leader_election.cc:343] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:29.560436 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f22a7b1b804661af3b7e785a833739" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:29.560724 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadb2ab1d2024f889841ac91e065e064" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:29.561583 1049 raft_consensus.cc:3054] T c69587c5a09d40f0baba5c2f6e8a7b41 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:29.562368 854 leader_election.cc:343] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:29.562506 854 leader_election.cc:304] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:29.562909 854 leader_election.cc:343] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:29.562976 1171 ts_tablet_manager.cc:1918] T f6803d0f553b4204b415b2734286bf4a P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:29.563074 854 leader_election.cc:304] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:29.563125 1171 log.cc:1198] T f6803d0f553b4204b415b2734286bf4a P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/f6803d0f553b4204b415b2734286bf4a
W20250112 02:52:29.563495 854 consensus_peers.cc:487] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:29.563591 1171 ts_tablet_manager.cc:1939] T f6803d0f553b4204b415b2734286bf4a P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:29.563755 1458 raft_consensus.cc:2743] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:29.563922 1458 raft_consensus.cc:2743] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:29.564743 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet f6803d0f553b4204b415b2734286bf4a (table default.Table_c9cd1883fedc45e89202fd92fbbd0e50 [id=944bc8032e5b4d32a6d568676f53a401]) successfully deleted
I20250112 02:52:29.566586 1171 tablet_replica.cc:331] T a1a0a13696fc41ccb28218f1275a4b5d P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:29.566668 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c686fdc8896463c81c9f7c5f53a3da2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:29.566859 1171 raft_consensus.cc:2238] T a1a0a13696fc41ccb28218f1275a4b5d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.567035 1171 raft_consensus.cc:2267] T a1a0a13696fc41ccb28218f1275a4b5d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.567812 1171 ts_tablet_manager.cc:1905] T a1a0a13696fc41ccb28218f1275a4b5d P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.569350 521 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_829049fe22b74e62a87dd1bed65fb3fe" } modify_external_catalogs: true
I20250112 02:52:29.569528 521 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_829049fe22b74e62a87dd1bed65fb3fe" } modify_external_catalogs: true
I20250112 02:52:29.569607 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_add1776e850b4a1bbd63f05707b51873"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:29.570036 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_31424b670f274fd3abd17eb8451ba95c" } modify_external_catalogs: true
W20250112 02:52:29.570138 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_add1776e850b4a1bbd63f05707b51873 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:29.570205 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_31424b670f274fd3abd17eb8451ba95c" } modify_external_catalogs: true
I20250112 02:52:29.571276 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_5d1f4404d0cc46268126e703afb52565" } new_table_name: "default.Table_f69a79e767cb4d6da20e31ddb7418a63" modify_external_catalogs: true
I20250112 02:52:29.571342 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_c816af31a1e945518fd886cbf8b8b885"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:29.571800 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c816af31a1e945518fd886cbf8b8b885 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:29.572502 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_3321055e22c3410eb5e5fe8be62e004c" } new_table_name: "default.Table_00028054ce7c431b96d17d8f6f8856a7" modify_external_catalogs: true
I20250112 02:52:29.573179 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_26dfe346379948b39f90b31653c2a463" } modify_external_catalogs: true
I20250112 02:52:29.573325 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_26dfe346379948b39f90b31653c2a463" } modify_external_catalogs: true
W20250112 02:52:29.575171 854 leader_election.cc:343] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:29.575323 854 leader_election.cc:304] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:29.576835 1171 ts_tablet_manager.cc:1918] T a1a0a13696fc41ccb28218f1275a4b5d P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:29.578974 1171 log.cc:1198] T a1a0a13696fc41ccb28218f1275a4b5d P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/a1a0a13696fc41ccb28218f1275a4b5d
I20250112 02:52:29.579826 1171 ts_tablet_manager.cc:1939] T a1a0a13696fc41ccb28218f1275a4b5d P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:29.580785 1030 tablet_service.cc:1467] Processing CreateTablet for tablet d63d411f906b4d4eb68343bdcc493e05 (DEFAULT_TABLE table=default.Table_e80769df10444dd5ae4bc23a8c6cc09e [id=d91a9f9e1f76482e99d4f10c0568f2e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.581132 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet a1a0a13696fc41ccb28218f1275a4b5d (table default.Table_62be2ee0d4744f198098621bf776cf01 [id=bf2df21352c64c069c2a06c878911016]) successfully deleted
I20250112 02:52:29.581161 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d84027ad6d40a5be5afcdd854431a5 with delete_type TABLET_DATA_DELETED (replaced by tablet fbe9e9624ede40c29ea852b2b5494965) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:29.581377 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 352c87f1dc724e7b94bc2e64abfce8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:29.581534 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d0f67141dc46b6a87ca0ad080d5bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:50844
I20250112 02:52:29.582247 1171 tablet_replica.cc:331] T afa2f8aabe5e42ccbc15b193fdc03563 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:29.582631 1171 raft_consensus.cc:2238] T afa2f8aabe5e42ccbc15b193fdc03563 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.582985 1171 raft_consensus.cc:2267] T afa2f8aabe5e42ccbc15b193fdc03563 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.583935 1171 ts_tablet_manager.cc:1905] T afa2f8aabe5e42ccbc15b193fdc03563 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.585469 1537 raft_consensus.cc:695] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:29.585764 1537 consensus_queue.cc:237] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.588243 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40934:
name: "default.Table_af9d530db1774d9eafed6727be07aeaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:29.588717 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af9d530db1774d9eafed6727be07aeaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:29.592032 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 5a3af0ea4e954053ae89814f3e74e6b7 (DEFAULT_TABLE table=default.Table_525750dad2b7461586d070dbaa74d0b1 [id=7d048aa907a74605964b68726ac5d759]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.587445 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 271) took 2576 ms. Trace:
I20250112 02:52:29.596119 1177 tablet_bootstrap.cc:492] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:29.596310 1177 ts_tablet_manager.cc:1397] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.090s user 0.002s sys 0.013s
I20250112 02:52:29.597076 1171 ts_tablet_manager.cc:1918] T afa2f8aabe5e42ccbc15b193fdc03563 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:29.591094 1457 raft_consensus.cc:2743] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:29.592500 1181 tablet_bootstrap.cc:492] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:29.597924 1171 log.cc:1198] T afa2f8aabe5e42ccbc15b193fdc03563 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/afa2f8aabe5e42ccbc15b193fdc03563
I20250112 02:52:29.598156 1181 ts_tablet_manager.cc:1397] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.396s user 0.006s sys 0.012s
I20250112 02:52:29.597052 1177 raft_consensus.cc:357] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.598502 1177 raft_consensus.cc:383] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.598439 1503 consensus_queue.cc:1035] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:29.598598 1177 raft_consensus.cc:738] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.598793 1177 consensus_queue.cc:260] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.599242 1177 ts_tablet_manager.cc:1428] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 02:52:29.599557 1177 tablet_bootstrap.cc:492] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:29.601400 1171 ts_tablet_manager.cc:1939] T afa2f8aabe5e42ccbc15b193fdc03563 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:29.593997 1175 rpcz_store.cc:276] 0112 02:52:27.010004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.010050 (+ 46us) service_pool.cc:225] Handling call
0112 02:52:29.163533 (+2153483us) ts_tablet_manager.cc:1145] Deleting tablet cb724feccc7f4fbc846fb6fa61be3385
0112 02:52:29.163545 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:29.170240 (+ 6695us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:29.172976 (+ 2736us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:29.586264 (+413288us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2153271}
I20250112 02:52:29.602514 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet afa2f8aabe5e42ccbc15b193fdc03563 (table default.Table_e21cbee8871f4ab583b22a3eaa4b5e6a [id=1b3ac0e8c0704c618077010dda4f7413]) successfully deleted
I20250112 02:52:29.603405 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet cb724feccc7f4fbc846fb6fa61be3385 (table default.Table_43048dfe7bf24df5a7254bdb6854101e [id=81a09e598ee34508aa88abfc6b44a2bd]) successfully deleted
I20250112 02:52:29.603657 1171 tablet_replica.cc:331] T 1726d61cc36a4deb83c5d604e8daf4f0 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:29.604281 1171 raft_consensus.cc:2238] T 1726d61cc36a4deb83c5d604e8daf4f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.604681 1171 raft_consensus.cc:2267] T 1726d61cc36a4deb83c5d604e8daf4f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.603966 1177 tablet_bootstrap.cc:654] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.606312 1171 ts_tablet_manager.cc:1905] T 1726d61cc36a4deb83c5d604e8daf4f0 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.603543 1181 raft_consensus.cc:357] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.609520 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 282) took 2389 ms. Trace:
I20250112 02:52:29.610512 1181 raft_consensus.cc:383] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.613112 1181 raft_consensus.cc:738] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.612092 1175 rpcz_store.cc:276] 0112 02:52:27.219202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.219368 (+ 166us) service_pool.cc:225] Handling call
0112 02:52:29.604898 (+2385530us) ts_tablet_manager.cc:1145] Deleting tablet 1a8a178bafcb457f9b0e9587df560493
0112 02:52:29.605283 (+ 385us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:29.608930 (+ 3647us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2368862}
I20250112 02:52:29.621927 1171 ts_tablet_manager.cc:1918] T 1726d61cc36a4deb83c5d604e8daf4f0 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:29.622130 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_27f1bff714d947d5b3f42be6bfbe8be2" } new_table_name: "default.Table_c16c321e8c3741838921ff50f04b8261" modify_external_catalogs: true
I20250112 02:52:29.623208 1171 log.cc:1198] T 1726d61cc36a4deb83c5d604e8daf4f0 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/1726d61cc36a4deb83c5d604e8daf4f0
I20250112 02:52:29.623646 1171 ts_tablet_manager.cc:1939] T 1726d61cc36a4deb83c5d604e8daf4f0 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:29.624157 743 tablet_service.cc:1467] Processing CreateTablet for tablet 709517ae520640208abe85fa14a5301e (DEFAULT_TABLE table=default.Table_b8af9fadb7b64c07afb49c07cf3ca5f9 [id=c7f00d3d40fb4a469c57e3f82983ff53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.624523 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 709517ae520640208abe85fa14a5301e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.624594 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 1726d61cc36a4deb83c5d604e8daf4f0 (table default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6]) successfully deleted
I20250112 02:52:29.624809 759 tablet_service.cc:1467] Processing CreateTablet for tablet 62f4bd2df0ce45dabc55373f1154d3ac (DEFAULT_TABLE table=default.Table_7921b99e24f84cceb58128f18925ad49 [id=0525f87eccbb4a1d8726d18a00f9a4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.625113 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f4bd2df0ce45dabc55373f1154d3ac. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:29.625142 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f776e2a6ec704cf9b64483bc7d5b1dff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f776e2a6ec704cf9b64483bc7d5b1dff already in progress: creating tablet
W20250112 02:52:29.625706 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d659c5a4f7a446c6aaf2283f65b45106 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d659c5a4f7a446c6aaf2283f65b45106 already in progress: creating tablet
I20250112 02:52:29.626142 1171 tablet_replica.cc:331] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:29.626389 1171 raft_consensus.cc:2238] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.626626 1171 raft_consensus.cc:2267] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.627103 1171 ts_tablet_manager.cc:1905] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.627753 747 tablet_service.cc:1467] Processing CreateTablet for tablet a3ffdc237f8640f0b378809f93fc6292 (DEFAULT_TABLE table=default.Table_fc264b6a06584c7ea757f5d18e05e2bc [id=b560a3488e3d4db58abc7d81d6094d86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.627791 745 tablet_service.cc:1467] Processing CreateTablet for tablet 54215c55966a46e1ab96e352c4e3764e (DEFAULT_TABLE table=default.Table_6e77252e817f43399c9b3f5b221795ad [id=1fa1474ab8804667b22e5f30c48c1630]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.624464 744 tablet_service.cc:1467] Processing CreateTablet for tablet 1a7717cdcb8a4f47b11ba7af54b57e3c (DEFAULT_TABLE table=default.Table_f7fd6e154c104f7ab66a01e7f8202da6 [id=4447b2d49c864e268d7eec91384899cc]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:29.628518 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 1a8a178bafcb457f9b0e9587df560493 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a8a178bafcb457f9b0e9587df560493 already in progress: creating tablet
I20250112 02:52:29.628578 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3ffdc237f8640f0b378809f93fc6292. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.629413 893 tablet_service.cc:1467] Processing CreateTablet for tablet 1a7717cdcb8a4f47b11ba7af54b57e3c (DEFAULT_TABLE table=default.Table_f7fd6e154c104f7ab66a01e7f8202da6 [id=4447b2d49c864e268d7eec91384899cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.630167 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a7717cdcb8a4f47b11ba7af54b57e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.630344 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a7717cdcb8a4f47b11ba7af54b57e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.630694 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54215c55966a46e1ab96e352c4e3764e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.630741 892 tablet_service.cc:1467] Processing CreateTablet for tablet a3ffdc237f8640f0b378809f93fc6292 (DEFAULT_TABLE table=default.Table_fc264b6a06584c7ea757f5d18e05e2bc [id=b560a3488e3d4db58abc7d81d6094d86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.631086 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3ffdc237f8640f0b378809f93fc6292. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.631863 887 tablet_service.cc:1467] Processing CreateTablet for tablet 62f4bd2df0ce45dabc55373f1154d3ac (DEFAULT_TABLE table=default.Table_7921b99e24f84cceb58128f18925ad49 [id=0525f87eccbb4a1d8726d18a00f9a4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.632174 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f4bd2df0ce45dabc55373f1154d3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.638736 1171 ts_tablet_manager.cc:1918] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:29.638908 1171 log.cc:1198] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/96e201d798c94e448a78e1c8adf41724
I20250112 02:52:29.639539 1171 ts_tablet_manager.cc:1939] T 96e201d798c94e448a78e1c8adf41724 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:29.641157 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 96e201d798c94e448a78e1c8adf41724 (table default.Table_4c59f5d61aab4b6184f0d9a4e21f965f [id=95cdab3e555643fc990b77b2ee8f058a]) successfully deleted
W20250112 02:52:29.642611 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d45fce86e02f46d4a57af718fa2a62c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d45fce86e02f46d4a57af718fa2a62c8 already in progress: creating tablet
W20250112 02:52:29.643499 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b6d84027ad6d40a5be5afcdd854431a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6d84027ad6d40a5be5afcdd854431a5 already in progress: creating tablet
W20250112 02:52:29.644323 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 352c87f1dc724e7b94bc2e64abfce8d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 352c87f1dc724e7b94bc2e64abfce8d7 already in progress: creating tablet
W20250112 02:52:29.645131 513 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 02d0f67141dc46b6a87ca0ad080d5bfe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02d0f67141dc46b6a87ca0ad080d5bfe already in progress: creating tablet
I20250112 02:52:29.646886 1458 raft_consensus.cc:491] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:29.647061 1458 raft_consensus.cc:513] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.647804 1458 leader_election.cc:290] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:29.648520 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9199b0d2b924a79938a979df4196bb4" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:29.648679 890 tablet_service.cc:1467] Processing CreateTablet for tablet 54215c55966a46e1ab96e352c4e3764e (DEFAULT_TABLE table=default.Table_6e77252e817f43399c9b3f5b221795ad [id=1fa1474ab8804667b22e5f30c48c1630]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.649029 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54215c55966a46e1ab96e352c4e3764e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:29.649397 855 leader_election.cc:343] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:29.652979 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 54215c55966a46e1ab96e352c4e3764e (DEFAULT_TABLE table=default.Table_6e77252e817f43399c9b3f5b221795ad [id=1fa1474ab8804667b22e5f30c48c1630]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.653542 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 62f4bd2df0ce45dabc55373f1154d3ac (DEFAULT_TABLE table=default.Table_7921b99e24f84cceb58128f18925ad49 [id=0525f87eccbb4a1d8726d18a00f9a4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.654047 1029 tablet_service.cc:1467] Processing CreateTablet for tablet a3ffdc237f8640f0b378809f93fc6292 (DEFAULT_TABLE table=default.Table_fc264b6a06584c7ea757f5d18e05e2bc [id=b560a3488e3d4db58abc7d81d6094d86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.656165 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 1a7717cdcb8a4f47b11ba7af54b57e3c (DEFAULT_TABLE table=default.Table_f7fd6e154c104f7ab66a01e7f8202da6 [id=4447b2d49c864e268d7eec91384899cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.657572 898 tablet_service.cc:1467] Processing CreateTablet for tablet 709517ae520640208abe85fa14a5301e (DEFAULT_TABLE table=default.Table_b8af9fadb7b64c07afb49c07cf3ca5f9 [id=c7f00d3d40fb4a469c57e3f82983ff53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.657919 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 709517ae520640208abe85fa14a5301e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.663053 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9199b0d2b924a79938a979df4196bb4" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:29.664057 854 leader_election.cc:343] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:29.664163 854 leader_election.cc:304] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:29.664649 1458 raft_consensus.cc:2743] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:29.618775 1181 consensus_queue.cc:260] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.628261 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 283) took 2403 ms. Trace:
I20250112 02:52:29.672681 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 709517ae520640208abe85fa14a5301e (DEFAULT_TABLE table=default.Table_b8af9fadb7b64c07afb49c07cf3ca5f9 [id=c7f00d3d40fb4a469c57e3f82983ff53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.686237 745 tablet_service.cc:1467] Processing CreateTablet for tablet fd81852962fb43449e35ef45dc1daaad (DEFAULT_TABLE table=default.Table_d45a3d1878fc47d5917bf66c670c5578 [id=8a4387d6a5084b85a8f28cb99798cec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.686713 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd81852962fb43449e35ef45dc1daaad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.669206 1175 rpcz_store.cc:276] 0112 02:52:27.223631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.223889 (+ 258us) service_pool.cc:225] Handling call
0112 02:52:29.619445 (+2395556us) ts_tablet_manager.cc:1145] Deleting tablet a85300a9aa1c4c50a3957a004bea7933
0112 02:52:29.619958 (+ 513us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:29.627010 (+ 7052us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2376823}
W20250112 02:52:29.694479 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet a85300a9aa1c4c50a3957a004bea7933 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a85300a9aa1c4c50a3957a004bea7933 already in progress: creating tablet
I20250112 02:52:29.696400 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 9b81aee21510452881948e41d7aba9be (DEFAULT_TABLE table=default.Table_29a6037a21724b5f9f0e0ee6a598a3dd [id=8b6988397e544a37b869f1e782095b64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.705997 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c639a2bcd4740e4be7b2b162004669a) from {username='slave'} at 127.0.0.1:40386
I20250112 02:52:29.706308 1171 tablet_replica.cc:331] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:29.706570 1171 raft_consensus.cc:2238] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.706755 1171 raft_consensus.cc:2267] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.707479 1171 ts_tablet_manager.cc:1905] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:29.712461 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 125b12a2f9164653be79d3e6d8313d36 has no consensus state
I20250112 02:52:29.712771 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_9c10574f7c7d435890eb9a24af23bcb9" } modify_external_catalogs: true
I20250112 02:52:29.712937 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40934:
table { table_name: "default.Table_9c10574f7c7d435890eb9a24af23bcb9" } modify_external_catalogs: true
I20250112 02:52:29.723912 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26577b614e32458e8bc5d6830c8c2e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.724171 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82b351c563d749df806ab5d0331a4ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.724411 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b81aee21510452881948e41d7aba9be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.726121 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3ffdc237f8640f0b378809f93fc6292. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.726318 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e1954e7859f422f9690fe4a9384a8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.726889 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f4bd2df0ce45dabc55373f1154d3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.727269 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea18666ec324403b9f10a099688438b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.727960 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cec5346e13164274ae7bc3b6aa241f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.731824 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41d5799b5b36487da42dc43b1c70ae26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.733954 1503 raft_consensus.cc:491] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:29.734112 1503 raft_consensus.cc:513] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.734862 1503 leader_election.cc:290] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:29.735816 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecfc7d8ba5e4406ebc61f37e6d500473" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:29.736367 714 leader_election.cc:343] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:29.739534 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu pid 492
I20250112 02:52:29.739744 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54215c55966a46e1ab96e352c4e3764e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.739946 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 709517ae520640208abe85fa14a5301e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.740222 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d63d411f906b4d4eb68343bdcc493e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.740676 556 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 02:52:29.743582 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03a32b424ec4431d88f6af45dd09089f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.763743 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a7717cdcb8a4f47b11ba7af54b57e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.767728 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a3af0ea4e954053ae89814f3e74e6b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.771304 1177 tablet_bootstrap.cc:492] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:29.771521 1177 ts_tablet_manager.cc:1397] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.172s user 0.042s sys 0.058s
I20250112 02:52:29.772637 1177 raft_consensus.cc:357] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.772899 1177 raft_consensus.cc:383] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.773008 1177 raft_consensus.cc:738] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.773298 1177 consensus_queue.cc:260] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.773902 1177 ts_tablet_manager.cc:1428] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:29.774168 1177 tablet_bootstrap.cc:492] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:29.774487 1171 ts_tablet_manager.cc:1918] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:29.775602 1503 raft_consensus.cc:491] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:29.777089 1171 log.cc:1198] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/2fdeaf5fa8cc4a2b97b2178d8ecbad69
I20250112 02:52:29.777112 1503 raft_consensus.cc:513] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.775785 1177 tablet_bootstrap.cc:654] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.777793 1503 leader_election.cc:290] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:29.778024 1171 ts_tablet_manager.cc:1939] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:29.778697 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d06e2e8c13464db8cf801a292bdc99" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:29.779934 714 leader_election.cc:343] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:29.781579 1028 tablet_service.cc:1467] Processing CreateTablet for tablet fd81852962fb43449e35ef45dc1daaad (DEFAULT_TABLE table=default.Table_d45a3d1878fc47d5917bf66c670c5578 [id=8a4387d6a5084b85a8f28cb99798cec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.785609 1177 tablet_bootstrap.cc:492] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:29.785799 1177 ts_tablet_manager.cc:1397] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20250112 02:52:29.786686 1177 raft_consensus.cc:357] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.786935 1177 raft_consensus.cc:383] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.787005 1177 raft_consensus.cc:738] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.787247 1177 consensus_queue.cc:260] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.787819 1177 ts_tablet_manager.cc:1428] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:29.788092 1177 tablet_bootstrap.cc:492] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:29.790081 1177 tablet_bootstrap.cc:654] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.813270 1177 tablet_bootstrap.cc:492] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:29.813392 1177 ts_tablet_manager.cc:1397] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.002s
I20250112 02:52:29.814380 1177 raft_consensus.cc:357] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.814553 1177 raft_consensus.cc:383] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.814633 1177 raft_consensus.cc:738] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.814812 1177 consensus_queue.cc:260] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.815387 1177 ts_tablet_manager.cc:1428] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:29.815780 1177 tablet_bootstrap.cc:492] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:29.817186 1177 tablet_bootstrap.cc:654] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.820411 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd81852962fb43449e35ef45dc1daaad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.824487 1177 tablet_bootstrap.cc:492] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:29.824627 1177 ts_tablet_manager.cc:1397] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250112 02:52:29.825376 1177 raft_consensus.cc:357] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.825578 1177 raft_consensus.cc:383] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.825649 1177 raft_consensus.cc:738] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.825850 1177 consensus_queue.cc:260] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.826344 1177 ts_tablet_manager.cc:1428] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:29.826627 1177 tablet_bootstrap.cc:492] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:29.826736 977 kernel_stack_watchdog.cc:198] Thread 1173 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:636 for 270ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f2e83341980 (unknown) at ??:0
@ 0x926cb7 __sanitizer::internal_sched_yield() at ??:?
@ 0x88c36d __interceptor_pthread_create at ??:?
@ 0x7f2e7ff51db1 kudu::Thread::StartThread() at ??:0
@ 0xa5e837 kudu::Thread::Create() at ??:?
@ 0x7f2e7ff727ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f2e7ff6e3bd kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f2e7ff734ff kudu::ThreadPool::Submit() at ??:0
@ 0x7f2e8b14dbcd kudu::log::SegmentAllocator::AsyncAllocateSegmentUnlocked() at ??:0
@ 0x7f2e8b14cbbf kudu::log::SegmentAllocator::AllocateSegmentAndRollOver() at ??:0
@ 0x7f2e8b14c790 kudu::log::SegmentAllocator::Init() at ??:0
@ 0x7f2e8b1546a5 kudu::log::Log::Init() at ??:0
@ 0x7f2e8b153a01 kudu::log::Log::Open() at ??:0
@ 0x7f2e8ca63e21 kudu::tablet::TabletBootstrap::OpenNewLog() at ??:0
@ 0x7f2e8ca6018c kudu::tablet::TabletBootstrap::RunBootstrap() at ??:0
@ 0x7f2e8ca5e90f kudu::tablet::TabletBootstrap::Bootstrap() at ??:0
I20250112 02:52:29.828069 1177 tablet_bootstrap.cc:654] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.830035 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 284) took 2534 ms. Trace:
I20250112 02:52:29.830461 1181 ts_tablet_manager.cc:1428] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.169s user 0.005s sys 0.001s
I20250112 02:52:29.830667 893 tablet_service.cc:1467] Processing CreateTablet for tablet fd81852962fb43449e35ef45dc1daaad (DEFAULT_TABLE table=default.Table_d45a3d1878fc47d5917bf66c670c5578 [id=8a4387d6a5084b85a8f28cb99798cec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:29.830866 1458 raft_consensus.cc:491] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:29.831322 1458 raft_consensus.cc:513] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.831931 1181 tablet_bootstrap.cc:492] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:29.832046 1458 leader_election.cc:290] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:29.832785 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb1a197d7e40689ccad82a44492371" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:29.834288 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb1a197d7e40689ccad82a44492371" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:29.830967 1175 rpcz_store.cc:276] 0112 02:52:27.235091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.235265 (+ 174us) service_pool.cc:225] Handling call
0112 02:52:29.694821 (+2459556us) ts_tablet_manager.cc:1145] Deleting tablet ac716c6da01941a989754aac1ef430d5
0112 02:52:29.694830 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:29.769658 (+ 74828us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2448139}
I20250112 02:52:29.835597 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecfc7d8ba5e4406ebc61f37e6d500473" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:29.836397 918 raft_consensus.cc:2463] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:29.835641 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d06e2e8c13464db8cf801a292bdc99" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:29.836736 917 raft_consensus.cc:2463] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:29.837005 1173 tablet_bootstrap.cc:492] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:29.837152 1173 ts_tablet_manager.cc:1397] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.441s user 0.042s sys 0.078s
W20250112 02:52:29.838749 855 leader_election.cc:343] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:29.838878 1173 raft_consensus.cc:357] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.839094 1173 raft_consensus.cc:383] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.839181 1173 raft_consensus.cc:738] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.839172 713 leader_election.cc:304] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:29.839556 1503 raft_consensus.cc:2798] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:29.839649 713 leader_election.cc:304] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:29.839828 1173 consensus_queue.cc:260] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.840075 1537 raft_consensus.cc:491] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:29.839681 1503 raft_consensus.cc:491] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:29.840204 1503 raft_consensus.cc:3054] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:29.840233 1537 raft_consensus.cc:513] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.840371 1172 ts_tablet_manager.cc:1918] T 96e201d798c94e448a78e1c8adf41724 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:29.840631 1172 log.cc:1198] T 96e201d798c94e448a78e1c8adf41724 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/96e201d798c94e448a78e1c8adf41724
I20250112 02:52:29.840875 1537 leader_election.cc:290] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:29.841055 1172 ts_tablet_manager.cc:1939] T 96e201d798c94e448a78e1c8adf41724 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
W20250112 02:52:29.841909 991 leader_election.cc:343] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: a1a0a13696fc41ccb28218f1275a4b5d
I20250112 02:52:29.842515 1173 ts_tablet_manager.cc:1428] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20250112 02:52:29.842749 1173 tablet_bootstrap.cc:492] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:29.842995 1503 raft_consensus.cc:513] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.844153 1173 tablet_bootstrap.cc:654] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.847795 1602 raft_consensus.cc:2798] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:29.847880 1602 raft_consensus.cc:491] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:29.847968 1602 raft_consensus.cc:3054] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:29.848105 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecfc7d8ba5e4406ebc61f37e6d500473" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:29.848793 1503 leader_election.cc:290] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:29.849004 714 leader_election.cc:343] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:29.850603 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 285) took 2600 ms. Trace:
I20250112 02:52:29.852339 1177 tablet_bootstrap.cc:492] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:29.852456 1177 ts_tablet_manager.cc:1397] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20250112 02:52:29.853071 1177 raft_consensus.cc:357] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:29.853227 1177 raft_consensus.cc:383] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.853307 1177 raft_consensus.cc:738] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.853530 1177 consensus_queue.cc:260] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:29.853956 1177 ts_tablet_manager.cc:1428] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:29.855023 1175 rpcz_store.cc:276] 0112 02:52:27.235899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.235953 (+ 54us) service_pool.cc:225] Handling call
0112 02:52:29.835039 (+2599086us) ts_tablet_manager.cc:1145] Deleting tablet 6cb76746b3a846c3aba531c04bf63966
0112 02:52:29.835049 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:29.835891 (+ 842us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2587586}
W20250112 02:52:29.855401 854 leader_election.cc:343] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:29.857731 1175 tablet_replica.cc:331] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:29.858001 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecfc7d8ba5e4406ebc61f37e6d500473" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:29.858161 917 raft_consensus.cc:3054] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:29.858455 854 leader_election.cc:304] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:29.858639 1175 raft_consensus.cc:2238] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.858799 1175 raft_consensus.cc:2267] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.859346 1175 ts_tablet_manager.cc:1905] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.860527 1458 raft_consensus.cc:2743] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:29.862179 1602 raft_consensus.cc:513] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.862735 1177 tablet_bootstrap.cc:492] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:29.864531 1177 tablet_bootstrap.cc:654] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:29.866823 989 leader_election.cc:343] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: a1a0a13696fc41ccb28218f1275a4b5d
I20250112 02:52:29.867693 1602 leader_election.cc:290] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:29.868238 989 leader_election.cc:304] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:29.868921 1537 raft_consensus.cc:2743] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:29.871776 1606 raft_consensus.cc:491] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f4a49a5756b143fcb767c3f41463f8d9)
I20250112 02:52:29.872016 1606 raft_consensus.cc:513] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:29.873755 1181 tablet_bootstrap.cc:654] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.873766 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d06e2e8c13464db8cf801a292bdc99" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:29.874809 918 raft_consensus.cc:3054] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:29.877664 918 raft_consensus.cc:2463] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:29.878307 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d06e2e8c13464db8cf801a292bdc99" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:29.878758 714 leader_election.cc:343] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:29.879225 713 leader_election.cc:304] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:29.879581 1602 raft_consensus.cc:2798] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:29.879742 1602 raft_consensus.cc:695] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:29.880051 1602 consensus_queue.cc:237] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.884176 917 raft_consensus.cc:2463] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:29.885636 713 leader_election.cc:304] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:29.885933 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd81852962fb43449e35ef45dc1daaad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:29.889673 1177 tablet_bootstrap.cc:492] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:29.889827 1503 raft_consensus.cc:2798] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:29.889851 1177 ts_tablet_manager.cc:1397] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.009s
I20250112 02:52:29.889971 1503 raft_consensus.cc:695] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:29.890239 1503 consensus_queue.cc:237] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.890616 1177 raft_consensus.cc:357] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.891265 1177 raft_consensus.cc:383] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.891345 1177 raft_consensus.cc:738] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.891619 1177 consensus_queue.cc:260] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.898736 1173 tablet_bootstrap.cc:492] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:29.899626 1177 ts_tablet_manager.cc:1428] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.010s user 0.003s sys 0.001s
I20250112 02:52:29.900013 1177 tablet_bootstrap.cc:492] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:29.905175 1173 ts_tablet_manager.cc:1397] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.005s
I20250112 02:52:29.905787 1173 raft_consensus.cc:357] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.905979 1173 raft_consensus.cc:383] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.906034 1173 raft_consensus.cc:738] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.914240 1175 ts_tablet_manager.cc:1918] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:29.914359 1175 log.cc:1198] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/1726d61cc36a4deb83c5d604e8daf4f0
I20250112 02:52:29.915104 1175 ts_tablet_manager.cc:1939] T 1726d61cc36a4deb83c5d604e8daf4f0 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:29.915800 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 288) took 2660 ms. Trace:
I20250112 02:52:29.915881 1175 rpcz_store.cc:276] 0112 02:52:27.255595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.260879 (+ 5284us) service_pool.cc:225] Handling call
0112 02:52:29.855316 (+2594437us) ts_tablet_manager.cc:1145] Deleting tablet 1726d61cc36a4deb83c5d604e8daf4f0
0112 02:52:29.855324 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:29.911541 (+ 56217us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:29.914143 (+ 2602us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:29.915737 (+ 1594us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2594203}
I20250112 02:52:29.916193 1175 tablet_replica.cc:331] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:29.916416 1175 raft_consensus.cc:2238] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.916561 1175 raft_consensus.cc:2267] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.930045 1177 tablet_bootstrap.cc:654] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.929981 464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd25f03f25df49178a5c53c636ecf304" candidate_term: 4 candidate_status { last_received { term: 3 index: 501 } } ignore_live_leader: false dest_uuid: "436220c23fa948efb5b6ca15875d8b0e" is_pre_election: true
I20250112 02:52:29.930745 1606 leader_election.cc:290] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 436220c23fa948efb5b6ca15875d8b0e (127.0.104.62:42341), f4a49a5756b143fcb767c3f41463f8d9 (127.0.104.61:38771)
I20250112 02:52:29.932410 1175 ts_tablet_manager.cc:1905] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.933430 1537 raft_consensus.cc:491] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:29.933562 1537 raft_consensus.cc:513] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.934114 1537 leader_election.cc:290] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:29.935200 991 leader_election.cc:343] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: f6803d0f553b4204b415b2734286bf4a
I20250112 02:52:29.937866 1175 ts_tablet_manager.cc:1918] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:29.937963 1175 log.cc:1198] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/afa2f8aabe5e42ccbc15b193fdc03563
I20250112 02:52:29.938794 1175 ts_tablet_manager.cc:1939] T afa2f8aabe5e42ccbc15b193fdc03563 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
W20250112 02:52:29.939368 989 leader_election.cc:343] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: f6803d0f553b4204b415b2734286bf4a
I20250112 02:52:29.939563 989 leader_election.cc:304] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:29.939985 1537 raft_consensus.cc:2743] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:29.907688 1173 consensus_queue.cc:260] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.940891 1173 ts_tablet_manager.cc:1428] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.036s user 0.000s sys 0.001s
I20250112 02:52:29.943774 1173 tablet_bootstrap.cc:492] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:29.945611 1173 tablet_bootstrap.cc:654] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.947152 1181 tablet_bootstrap.cc:492] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:29.947567 1181 ts_tablet_manager.cc:1397] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.117s user 0.006s sys 0.009s
I20250112 02:52:29.948752 1181 raft_consensus.cc:357] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.948966 1181 raft_consensus.cc:383] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.949034 1181 raft_consensus.cc:738] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.949267 1181 consensus_queue.cc:260] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.951676 1177 tablet_bootstrap.cc:492] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:29.952421 1181 ts_tablet_manager.cc:1428] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 02:52:29.952670 1181 tablet_bootstrap.cc:492] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:29.953485 1177 ts_tablet_manager.cc:1397] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.004s
I20250112 02:52:29.954155 1181 tablet_bootstrap.cc:654] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.954238 1177 raft_consensus.cc:357] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.954418 1177 raft_consensus.cc:383] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.954493 1177 raft_consensus.cc:738] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
W20250112 02:52:29.954576 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 68948.8us
I20250112 02:52:29.954670 1177 consensus_queue.cc:260] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.960487 1173 tablet_bootstrap.cc:492] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:29.960609 1173 ts_tablet_manager.cc:1397] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.002s
I20250112 02:52:29.961241 1173 raft_consensus.cc:357] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.961413 1173 raft_consensus.cc:383] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.961484 1173 raft_consensus.cc:738] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.961697 1173 consensus_queue.cc:260] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.962162 1173 ts_tablet_manager.cc:1428] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 02:52:29.962849 1173 tablet_bootstrap.cc:492] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:29.963726 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 287) took 2716 ms. Trace:
I20250112 02:52:29.963817 1175 rpcz_store.cc:276] 0112 02:52:27.247331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.250108 (+ 2777us) service_pool.cc:225] Handling call
0112 02:52:29.916120 (+2666012us) ts_tablet_manager.cc:1145] Deleting tablet afa2f8aabe5e42ccbc15b193fdc03563
0112 02:52:29.916128 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:29.935098 (+ 18970us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:29.937770 (+ 2672us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:29.963698 (+ 25928us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2627491}
I20250112 02:52:29.964145 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 286) took 2717 ms. Trace:
I20250112 02:52:29.965603 1173 tablet_bootstrap.cc:654] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.965849 1175 rpcz_store.cc:276] 0112 02:52:27.246235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.246506 (+ 271us) service_pool.cc:225] Handling call
0112 02:52:29.964034 (+2717528us) ts_tablet_manager.cc:1145] Deleting tablet c8a7281c0c09442ebd0cc13b8496c86e
0112 02:52:29.964041 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:29.964121 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2675154}
I20250112 02:52:29.966183 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 290) took 2637 ms. Trace:
I20250112 02:52:29.966327 1175 rpcz_store.cc:276] 0112 02:52:27.328864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.347512 (+ 18648us) service_pool.cc:225] Handling call
0112 02:52:29.966070 (+2618558us) ts_tablet_manager.cc:1145] Deleting tablet 015412a5fa304453bd8d1be1c3cd7a46
0112 02:52:29.966077 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:29.966153 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2618247}
I20250112 02:52:29.966575 1175 tablet_replica.cc:331] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:29.966758 1175 raft_consensus.cc:2238] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:29.966897 1175 raft_consensus.cc:2267] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:29.967705 1177 ts_tablet_manager.cc:1428] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20250112 02:52:29.968034 1177 tablet_bootstrap.cc:492] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:29.969022 1181 tablet_bootstrap.cc:492] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:29.969128 1181 ts_tablet_manager.cc:1397] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20250112 02:52:29.969790 1181 raft_consensus.cc:357] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.969969 1181 raft_consensus.cc:383] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.970048 1181 raft_consensus.cc:738] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.970259 1181 consensus_queue.cc:260] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.973080 1181 ts_tablet_manager.cc:1428] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 02:52:29.973330 1181 tablet_bootstrap.cc:492] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:29.974943 1181 tablet_bootstrap.cc:654] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.976804 1175 ts_tablet_manager.cc:1905] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:29.983004 1175 ts_tablet_manager.cc:1918] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:29.983129 1175 log.cc:1198] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/2fdeaf5fa8cc4a2b97b2178d8ecbad69
I20250112 02:52:29.984074 1173 tablet_bootstrap.cc:492] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:29.984715 1173 ts_tablet_manager.cc:1397] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.001s
I20250112 02:52:29.985545 1173 raft_consensus.cc:357] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.985738 1173 raft_consensus.cc:383] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.985792 1173 raft_consensus.cc:738] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.986016 1173 consensus_queue.cc:260] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.986555 1173 ts_tablet_manager.cc:1428] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:29.987303 1177 tablet_bootstrap.cc:654] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.991743 1173 tablet_bootstrap.cc:492] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:29.993758 1173 tablet_bootstrap.cc:654] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:29.996474 1173 tablet_bootstrap.cc:492] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:29.996593 1173 ts_tablet_manager.cc:1397] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20250112 02:52:29.997237 1173 raft_consensus.cc:357] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.997450 1173 raft_consensus.cc:383] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:29.997531 1173 raft_consensus.cc:738] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:29.997713 1173 consensus_queue.cc:260] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:29.998127 1173 ts_tablet_manager.cc:1428] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 02:52:29.998382 1173 tablet_bootstrap.cc:492] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:30.000128 1175 ts_tablet_manager.cc:1939] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:30.000924 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 291) took 2594 ms. Trace:
I20250112 02:52:30.001025 1175 rpcz_store.cc:276] 0112 02:52:27.406035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.406089 (+ 54us) service_pool.cc:225] Handling call
0112 02:52:29.966521 (+2560432us) ts_tablet_manager.cc:1145] Deleting tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69
0112 02:52:29.966528 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:29.980045 (+ 13517us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:29.982920 (+ 2875us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.000854 (+ 17934us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2560206}
I20250112 02:52:30.001365 1175 tablet_replica.cc:331] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:30.001536 1175 raft_consensus.cc:2238] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:30.004403 1177 tablet_bootstrap.cc:492] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.004668 1177 ts_tablet_manager.cc:1397] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.002s
I20250112 02:52:30.009100 1173 tablet_bootstrap.cc:654] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.008131 1181 tablet_bootstrap.cc:492] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:30.201719 1181 ts_tablet_manager.cc:1397] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.228s user 0.002s sys 0.003s
W20250112 02:52:30.183173 511 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet bb4ef0220dde43569452379639759db5 (table default.Table_f7421d0779924239a3b493954ef748b7 [id=b42fa8fc65e344f989a98ae411e916ed], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bb4ef0220dde43569452379639759db5 consensus state has no leader
I20250112 02:52:30.190773 513 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 (table default.Table_0dec5de812ab4ea48f1b7a8f4cb0cc4f [id=2f43ebd749304c0287769224ac6b12c7]) successfully deleted
I20250112 02:52:30.202375 1181 raft_consensus.cc:357] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.202572 1181 raft_consensus.cc:383] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.192696 512 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 96e201d798c94e448a78e1c8adf41724 (table default.Table_4c59f5d61aab4b6184f0d9a4e21f965f [id=95cdab3e555643fc990b77b2ee8f058a]) successfully deleted
I20250112 02:52:30.202622 1181 raft_consensus.cc:738] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
W20250112 02:52:30.202724 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet ac716c6da01941a989754aac1ef430d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac716c6da01941a989754aac1ef430d5 already in progress: creating tablet
I20250112 02:52:30.202813 1181 consensus_queue.cc:260] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:30.203016 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 6cb76746b3a846c3aba531c04bf63966 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6cb76746b3a846c3aba531c04bf63966 already in progress: creating tablet
I20250112 02:52:30.203241 1181 ts_tablet_manager.cc:1428] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 02:52:30.203361 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 1726d61cc36a4deb83c5d604e8daf4f0 (table default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6]) successfully deleted
I20250112 02:52:30.203567 1181 tablet_bootstrap.cc:492] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:30.203630 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet afa2f8aabe5e42ccbc15b193fdc03563 (table default.Table_e21cbee8871f4ab583b22a3eaa4b5e6a [id=1b3ac0e8c0704c618077010dda4f7413]) successfully deleted
W20250112 02:52:30.203809 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet c8a7281c0c09442ebd0cc13b8496c86e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8a7281c0c09442ebd0cc13b8496c86e already in progress: creating tablet
W20250112 02:52:30.203977 511 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 015412a5fa304453bd8d1be1c3cd7a46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 015412a5fa304453bd8d1be1c3cd7a46 already in progress: creating tablet
I20250112 02:52:30.204131 511 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 (table default.Table_0dec5de812ab4ea48f1b7a8f4cb0cc4f [id=2f43ebd749304c0287769224ac6b12c7]) successfully deleted
I20250112 02:52:30.005489 1177 raft_consensus.cc:357] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.204368 1177 raft_consensus.cc:383] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.204460 1177 raft_consensus.cc:738] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.204723 1177 consensus_queue.cc:260] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.205184 1181 tablet_bootstrap.cc:654] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.205189 1177 ts_tablet_manager.cc:1428] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.200s user 0.000s sys 0.002s
I20250112 02:52:30.205271 892 tablet_service.cc:1467] Processing CreateTablet for tablet 896908b2726c40f9aabdfc9a6b3283f7 (DEFAULT_TABLE table=default.Table_feaebd2f4cbb47b6b24ff79614994d6f [id=63c20c66463d4a30b7e1558fecce2704]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:30.205507 1177 tablet_bootstrap.cc:492] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:30.008497 1175 raft_consensus.cc:2267] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:30.128512 1602 raft_consensus.cc:491] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.206130 745 tablet_service.cc:1467] Processing CreateTablet for tablet 896908b2726c40f9aabdfc9a6b3283f7 (DEFAULT_TABLE table=default.Table_feaebd2f4cbb47b6b24ff79614994d6f [id=63c20c66463d4a30b7e1558fecce2704]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:30.206338 1602 raft_consensus.cc:513] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.206579 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 896908b2726c40f9aabdfc9a6b3283f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:30.207043 1602 leader_election.cc:290] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:30.207100 1177 tablet_bootstrap.cc:654] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.020613 1537 raft_consensus.cc:491] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.162473 1634 raft_consensus.cc:491] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.207598 1537 raft_consensus.cc:513] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.207746 1634 raft_consensus.cc:513] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.208228 1537 leader_election.cc:290] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.208321 1634 leader_election.cc:290] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.208563 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2e31a65294407bb62b13e92898a7d2" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:30.208999 714 leader_election.cc:343] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.200516 536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd25f03f25df49178a5c53c636ecf304" candidate_term: 4 candidate_status { last_received { term: 3 index: 501 } } ignore_live_leader: false dest_uuid: "f4a49a5756b143fcb767c3f41463f8d9" is_pre_election: true
I20250112 02:52:30.161867 1611 raft_consensus.cc:491] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.186163 759 tablet_service.cc:1467] Processing CreateTablet for tablet 9b81aee21510452881948e41d7aba9be (DEFAULT_TABLE table=default.Table_29a6037a21724b5f9f0e0ee6a598a3dd [id=8b6988397e544a37b869f1e782095b64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:30.209899 1611 raft_consensus.cc:513] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.210095 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b81aee21510452881948e41d7aba9be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:30.210429 744 tablet_service.cc:1467] Processing CreateTablet for tablet ca6f080e5fd84084b601b646d4e6c9e2 (DEFAULT_TABLE table=default.Table_613140584042445b9d71a3d05cee62c1 [id=f95ac050a4b840a19a8175caade87d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:30.210618 1611 leader_election.cc:290] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.210835 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca6f080e5fd84084b601b646d4e6c9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:30.210871 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu with pid 492
I20250112 02:52:30.211753 747 tablet_service.cc:1467] Processing CreateTablet for tablet 164f76edcf8f4bd498dcd2c25fd7477d (DEFAULT_TABLE table=default.Table_5e6ad2a2040d41b381bb2653c6f691ec [id=a2024dc6a87341cb87c45eee7ab024b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:30.212078 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 164f76edcf8f4bd498dcd2c25fd7477d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:30.215749 743 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9280db29e54d41af1bed2fa5aa5535 (DEFAULT_TABLE table=default.Table_6671f628f89b428a83eca77e9ca26516 [id=e0577e1a40064437bb7eb4c66fbd914e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:30.215900 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7bd96fb381f4e73a0081be3fc799c36" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:30.216149 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b9280db29e54d41af1bed2fa5aa5535. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:30.216297 714 leader_election.cc:343] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.219813 760 tablet_service.cc:1467] Processing CreateTablet for tablet 05000c6a48e0450d8e6685ca864f67f0 (DEFAULT_TABLE table=default.Table_692d9e1bfcfd435e8c420696c1e5aae5 [id=1ec97dc2b50a4a3dbbc449559938cead]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:30.220089 1173 tablet_bootstrap.cc:492] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.220235 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05000c6a48e0450d8e6685ca864f67f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:30.220240 1173 ts_tablet_manager.cc:1397] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.222s user 0.000s sys 0.005s
I20250112 02:52:30.220963 1173 raft_consensus.cc:357] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.221148 1173 raft_consensus.cc:383] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.221200 1173 raft_consensus.cc:738] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.221408 1173 consensus_queue.cc:260] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.223639 746 tablet_service.cc:1467] Processing CreateTablet for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (DEFAULT_TABLE table=default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:30.224035 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7340a7a916c4b2fab32e0ba2d0ce322. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:30.224828 1177 tablet_bootstrap.cc:492] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.225107 1177 ts_tablet_manager.cc:1397] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.008s
I20250112 02:52:30.225209 1173 ts_tablet_manager.cc:1428] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 02:52:30.225965 1177 raft_consensus.cc:357] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.226130 1177 raft_consensus.cc:383] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.226194 1177 raft_consensus.cc:738] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.226397 1177 consensus_queue.cc:260] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.227030 1177 ts_tablet_manager.cc:1428] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:30.227285 1177 tablet_bootstrap.cc:492] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:30.227634 1173 tablet_bootstrap.cc:492] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:30.227707 758 tablet_service.cc:1467] Processing CreateTablet for tablet 5804fc4130f84c69832c1022b7bd7fa5 (DEFAULT_TABLE table=default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:30.227982 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5804fc4130f84c69832c1022b7bd7fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:30.229771 1177 tablet_bootstrap.cc:654] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.230252 1173 tablet_bootstrap.cc:654] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.231768 757 tablet_service.cc:1467] Processing CreateTablet for tablet 24ebf85f76f44633a20d97964638c0ac (DEFAULT_TABLE table=default.Table_849b7f79a5784f5fabed0136deb0a129 [id=386e3565dbf2490997e5cc81d6f02567]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:30.232141 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ebf85f76f44633a20d97964638c0ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:30.235452 1634 raft_consensus.cc:491] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.235648 1634 raft_consensus.cc:513] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.236177 1634 leader_election.cc:290] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.238004 1611 raft_consensus.cc:491] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.238137 1611 raft_consensus.cc:513] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.238692 1611 leader_election.cc:290] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:30.238755 889 tablet_service.cc:1467] Processing CreateTablet for tablet 5804fc4130f84c69832c1022b7bd7fa5 (DEFAULT_TABLE table=default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:30.239897 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c530aa0aa64e47ae27d61f341d2bd9" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:30.240475 887 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9280db29e54d41af1bed2fa5aa5535 (DEFAULT_TABLE table=default.Table_6671f628f89b428a83eca77e9ca26516 [id=e0577e1a40064437bb7eb4c66fbd914e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:30.241060 714 leader_election.cc:343] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:30.241036 991 leader_election.cc:343] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: a47cdc208fbc431b8326364e84432cf5
W20250112 02:52:30.241238 991 leader_election.cc:343] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: 97b31ebe035c4552baaffc3133ec73c1
I20250112 02:52:30.241886 1175 ts_tablet_manager.cc:1905] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:30.248549 1173 tablet_bootstrap.cc:492] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.248652 1173 ts_tablet_manager.cc:1397] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20250112 02:52:30.241850 899 tablet_service.cc:1467] Processing CreateTablet for tablet 05000c6a48e0450d8e6685ca864f67f0 (DEFAULT_TABLE table=default.Table_692d9e1bfcfd435e8c420696c1e5aae5 [id=1ec97dc2b50a4a3dbbc449559938cead]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:30.242280 895 tablet_service.cc:1467] Processing CreateTablet for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (DEFAULT_TABLE table=default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:30.249226 891 tablet_service.cc:1467] Processing CreateTablet for tablet 24ebf85f76f44633a20d97964638c0ac (DEFAULT_TABLE table=default.Table_849b7f79a5784f5fabed0136deb0a129 [id=386e3565dbf2490997e5cc81d6f02567]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:30.249632 893 tablet_service.cc:1467] Processing CreateTablet for tablet 9b81aee21510452881948e41d7aba9be (DEFAULT_TABLE table=default.Table_29a6037a21724b5f9f0e0ee6a598a3dd [id=8b6988397e544a37b869f1e782095b64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:30.249979 1173 raft_consensus.cc:357] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.250409 1173 raft_consensus.cc:383] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.250591 1173 raft_consensus.cc:738] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
W20250112 02:52:30.251812 991 leader_election.cc:343] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: b2d8ea2ef18a46508371b4789ab6f4fa
I20250112 02:52:30.252207 898 tablet_service.cc:1467] Processing CreateTablet for tablet 164f76edcf8f4bd498dcd2c25fd7477d (DEFAULT_TABLE table=default.Table_5e6ad2a2040d41b381bb2653c6f691ec [id=a2024dc6a87341cb87c45eee7ab024b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:30.252383 890 tablet_service.cc:1467] Processing CreateTablet for tablet ca6f080e5fd84084b601b646d4e6c9e2 (DEFAULT_TABLE table=default.Table_613140584042445b9d71a3d05cee62c1 [id=f95ac050a4b840a19a8175caade87d48]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:30.252959 989 leader_election.cc:343] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: a47cdc208fbc431b8326364e84432cf5
I20250112 02:52:30.253106 989 leader_election.cc:304] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
W20250112 02:52:30.253494 989 leader_election.cc:343] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 97b31ebe035c4552baaffc3133ec73c1
I20250112 02:52:30.253621 989 leader_election.cc:304] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:30.253999 1634 raft_consensus.cc:2743] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.254164 1634 raft_consensus.cc:2743] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.254348 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 896908b2726c40f9aabdfc9a6b3283f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:30.256264 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2e31a65294407bb62b13e92898a7d2" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:30.250981 1173 consensus_queue.cc:260] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.261785 1173 ts_tablet_manager.cc:1428] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250112 02:52:30.263828 917 raft_consensus.cc:2463] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:30.263840 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b9280db29e54d41af1bed2fa5aa5535. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:30.264464 713 leader_election.cc:304] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.264580 1173 tablet_bootstrap.cc:492] T ecfc7d8ba5e4406ebc61f37e6d500473 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:30.264895 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7bd96fb381f4e73a0081be3fc799c36" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:30.265107 918 raft_consensus.cc:2463] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:30.266814 1611 raft_consensus.cc:2798] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:30.266940 1611 raft_consensus.cc:491] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250112 02:52:30.268061 989 leader_election.cc:343] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: b2d8ea2ef18a46508371b4789ab6f4fa
I20250112 02:52:30.269377 1611 raft_consensus.cc:3054] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.272406 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7340a7a916c4b2fab32e0ba2d0ce322. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:30.275511 989 leader_election.cc:304] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:30.275969 1537 raft_consensus.cc:2743] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.277648 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca6f080e5fd84084b601b646d4e6c9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:30.279397 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b81aee21510452881948e41d7aba9be. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:30.285535 855 consensus_peers.cc:487] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96e201d798c94e448a78e1c8adf41724. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:30.287657 1173 tablet_bootstrap.cc:654] T ecfc7d8ba5e4406ebc61f37e6d500473 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.289175 713 leader_election.cc:304] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.290380 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ebf85f76f44633a20d97964638c0ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:30.292946 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 164f76edcf8f4bd498dcd2c25fd7477d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:30.294697 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5804fc4130f84c69832c1022b7bd7fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:30.295444 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05000c6a48e0450d8e6685ca864f67f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:30.298002 1609 raft_consensus.cc:2798] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:30.298108 1609 raft_consensus.cc:491] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250112 02:52:30.299353 713 leader_election.cc:343] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: c7c530aa0aa64e47ae27d61f341d2bd9
I20250112 02:52:30.300590 1173 tablet_bootstrap.cc:492] T ecfc7d8ba5e4406ebc61f37e6d500473 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.300593 713 leader_election.cc:304] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:30.300926 1602 raft_consensus.cc:2743] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.301301 1609 raft_consensus.cc:3054] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.301517 1173 ts_tablet_manager.cc:1397] T ecfc7d8ba5e4406ebc61f37e6d500473 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.002s
I20250112 02:52:30.302299 1173 raft_consensus.cc:357] T ecfc7d8ba5e4406ebc61f37e6d500473 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.302534 1173 raft_consensus.cc:383] T ecfc7d8ba5e4406ebc61f37e6d500473 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.302614 1173 raft_consensus.cc:738] T ecfc7d8ba5e4406ebc61f37e6d500473 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.302839 1173 consensus_queue.cc:260] T ecfc7d8ba5e4406ebc61f37e6d500473 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.304468 1602 raft_consensus.cc:491] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.304666 1602 raft_consensus.cc:513] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.306914 1602 leader_election.cc:290] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.307201 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd498ccce484a89855c982762086890" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:30.307798 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd498ccce484a89855c982762086890" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:30.307982 919 raft_consensus.cc:2463] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
W20250112 02:52:30.308431 990 connection.cc:537] server connection from 127.0.0.1:50844 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:52:30.309265 1345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.61:38771 (error 108)
W20250112 02:52:30.310146 714 leader_election.cc:343] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:30.310946 1107 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.61:38771)
W20250112 02:52:30.311508 1107 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.61:38771)
W20250112 02:52:30.311995 1107 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.61:38771)
W20250112 02:52:30.312659 1107 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.61:38771)
W20250112 02:52:30.323696 440 connection.cc:537] server connection from 127.0.0.1:40576 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 02:52:30.325736 1611 raft_consensus.cc:513] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:30.331422 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51456 (request call id 411) could send its response
W20250112 02:52:30.334579 1649 negotiation.cc:337] Failed RPC negotiation. Trace:
0112 02:52:30.315375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:38771 (local address 127.0.104.2:58835)
0112 02:52:30.333986 (+ 18611us) negotiation.cc:107] Waiting for socket to connect
0112 02:52:30.334133 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":18469,"threads_started":1}
W20250112 02:52:30.335865 1648 negotiation.cc:337] Failed RPC negotiation. Trace:
0112 02:52:30.314953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:38771 (local address 127.0.0.1:43806)
0112 02:52:30.329834 (+ 14881us) negotiation.cc:107] Waiting for socket to connect
0112 02:52:30.330040 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":14718,"thread_start_us":14638,"threads_started":1}
W20250112 02:52:30.342375 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51456 (request call id 409) could send its response
W20250112 02:52:30.344662 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51456 (request call id 405) could send its response
I20250112 02:52:30.346781 1173 ts_tablet_manager.cc:1428] T ecfc7d8ba5e4406ebc61f37e6d500473 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.045s user 0.001s sys 0.001s
I20250112 02:52:30.352982 1173 tablet_bootstrap.cc:492] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:30.356196 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2e31a65294407bb62b13e92898a7d2" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:30.356783 1173 tablet_bootstrap.cc:654] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:30.352487 713 connection.cc:537] server connection from 127.0.0.1:40386 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:52:30.358724 1345 leader_election.cc:336] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f4a49a5756b143fcb767c3f41463f8d9 (127.0.104.61:38771): Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection refused (error 111)
I20250112 02:52:30.358997 1345 leader_election.cc:304] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd25f03f25df49178a5c53c636ecf304; no voters: 436220c23fa948efb5b6ca15875d8b0e, f4a49a5756b143fcb767c3f41463f8d9
I20250112 02:52:30.359549 1606 raft_consensus.cc:2743] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250112 02:52:30.360759 1644 negotiation.cc:337] Failed RPC negotiation. Trace:
0112 02:52:30.308880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:38771 (local address 127.0.104.3:56361)
0112 02:52:30.360221 (+ 51341us) negotiation.cc:107] Waiting for socket to connect
0112 02:52:30.360402 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":51205,"thread_start_us":51123,"threads_started":1}
I20250112 02:52:30.360980 1173 tablet_bootstrap.cc:492] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.361438 1173 ts_tablet_manager.cc:1397] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
W20250112 02:52:30.361838 714 leader_election.cc:343] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.362650 1173 raft_consensus.cc:357] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:30.362015 713 connection.cc:441] server connection from 127.0.0.1:40386 torn down before Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40386 (request call id 399) could send its response
I20250112 02:52:30.363483 1173 raft_consensus.cc:383] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.363734 1173 raft_consensus.cc:738] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.363729 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2e31a65294407bb62b13e92898a7d2" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:30.363979 916 raft_consensus.cc:3054] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:30.364161 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51456 (request call id 407) could send its response
I20250112 02:52:30.364099 1173 consensus_queue.cc:260] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:30.365401 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51456 (request call id 412) could send its response
W20250112 02:52:30.365871 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51456 (request call id 410) could send its response
I20250112 02:52:30.366729 1173 ts_tablet_manager.cc:1428] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250112 02:52:30.366719 916 raft_consensus.cc:2463] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:30.367004 1173 tablet_bootstrap.cc:492] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:30.371402 1181 tablet_bootstrap.cc:492] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:30.371537 1181 ts_tablet_manager.cc:1397] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.168s user 0.008s sys 0.018s
I20250112 02:52:30.372198 1181 raft_consensus.cc:357] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.372385 1181 raft_consensus.cc:383] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.372467 1181 raft_consensus.cc:738] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.372666 1181 consensus_queue.cc:260] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.372927 1173 tablet_bootstrap.cc:654] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.373103 1181 ts_tablet_manager.cc:1428] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:30.373345 1181 tablet_bootstrap.cc:492] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:30.375062 1181 tablet_bootstrap.cc:654] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.381062 1173 tablet_bootstrap.cc:492] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.381623 1173 ts_tablet_manager.cc:1397] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.006s
W20250112 02:52:30.381667 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51456 (request call id 413) could send its response
I20250112 02:52:30.382441 1173 raft_consensus.cc:357] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.382761 1181 tablet_bootstrap.cc:492] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:30.382936 1181 ts_tablet_manager.cc:1397] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20250112 02:52:30.383607 1173 raft_consensus.cc:383] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.383823 1173 raft_consensus.cc:738] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.383740 1181 raft_consensus.cc:357] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.383996 1181 raft_consensus.cc:383] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.384078 1181 raft_consensus.cc:738] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.384317 1181 consensus_queue.cc:260] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.384720 1181 ts_tablet_manager.cc:1428] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:30.384315 1173 consensus_queue.cc:260] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.384984 1181 tablet_bootstrap.cc:492] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:30.385684 1175 ts_tablet_manager.cc:1918] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:30.385794 1175 log.cc:1198] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/6b521522c4784d2ab4056268823f1099
I20250112 02:52:30.386128 1175 ts_tablet_manager.cc:1939] T 6b521522c4784d2ab4056268823f1099 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:30.387152 1173 ts_tablet_manager.cc:1428] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250112 02:52:30.387887 1173 tablet_bootstrap.cc:492] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:30.387650 1181 tablet_bootstrap.cc:654] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.388383 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 295) took 2924 ms. Trace:
I20250112 02:52:30.389361 1175 rpcz_store.cc:276] 0112 02:52:27.464133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.464198 (+ 65us) service_pool.cc:225] Handling call
0112 02:52:30.001275 (+2537077us) ts_tablet_manager.cc:1145] Deleting tablet 6b521522c4784d2ab4056268823f1099
0112 02:52:30.001291 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.265062 (+263771us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.385575 (+120513us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.388344 (+ 2769us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2536872}
W20250112 02:52:30.389653 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 295) could send its response
I20250112 02:52:30.389731 1175 tablet_replica.cc:331] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:30.389935 1175 raft_consensus.cc:2238] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:30.390298 1175 raft_consensus.cc:2267] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:30.390882 1175 ts_tablet_manager.cc:1905] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:30.392433 1173 tablet_bootstrap.cc:654] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:30.392441 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51456 (request call id 408) could send its response
I20250112 02:52:30.397300 1175 ts_tablet_manager.cc:1918] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:30.397470 1175 log.cc:1198] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/5a8e1b96accd459eaed221ded4a453b6
I20250112 02:52:30.398102 1175 ts_tablet_manager.cc:1939] T 5a8e1b96accd459eaed221ded4a453b6 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:30.398651 1611 leader_election.cc:290] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:30.399031 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 297) took 2910 ms. Trace:
I20250112 02:52:30.399150 1175 rpcz_store.cc:276] 0112 02:52:27.488503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.488651 (+ 148us) service_pool.cc:225] Handling call
0112 02:52:30.389671 (+2901020us) ts_tablet_manager.cc:1145] Deleting tablet 5a8e1b96accd459eaed221ded4a453b6
0112 02:52:30.389681 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.393797 (+ 4116us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.397183 (+ 3386us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.398949 (+ 1766us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2900832}
W20250112 02:52:30.399497 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 297) could send its response
I20250112 02:52:30.399688 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 298) took 2911 ms. Trace:
I20250112 02:52:30.399803 1175 rpcz_store.cc:276] 0112 02:52:27.488613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.488859 (+ 246us) service_pool.cc:225] Handling call
0112 02:52:30.399504 (+2910645us) ts_tablet_manager.cc:1145] Deleting tablet 1b184806b4884cd3b0a002905a1e57af
0112 02:52:30.399514 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.399619 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2910517}
W20250112 02:52:30.400115 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 298) could send its response
W20250112 02:52:30.399999 713 connection.cc:441] server connection from 127.0.0.1:40386 torn down before Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40386 (request call id 401) could send its response
I20250112 02:52:30.401053 713 leader_election.cc:304] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.401335 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 299) took 2910 ms. Trace:
I20250112 02:52:30.401441 1175 rpcz_store.cc:276] 0112 02:52:27.490994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.491042 (+ 48us) service_pool.cc:225] Handling call
0112 02:52:30.401126 (+2910084us) ts_tablet_manager.cc:1145] Deleting tablet e5c10fad790046a88fbc8ebe0303c481
0112 02:52:30.401138 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.401278 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2909906}
I20250112 02:52:30.401539 1611 raft_consensus.cc:2798] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
W20250112 02:52:30.401705 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 299) could send its response
I20250112 02:52:30.401844 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 300) took 2908 ms. Trace:
I20250112 02:52:30.401957 1175 rpcz_store.cc:276] 0112 02:52:27.493329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.493375 (+ 46us) service_pool.cc:225] Handling call
0112 02:52:30.401714 (+2908339us) ts_tablet_manager.cc:1145] Deleting tablet 7a34d03a80ae402db6607261a5c40b60
0112 02:52:30.401723 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.401790 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2907188}
W20250112 02:52:30.402240 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 300) could send its response
I20250112 02:52:30.402375 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 301) took 2874 ms. Trace:
I20250112 02:52:30.402482 1175 rpcz_store.cc:276] 0112 02:52:27.528356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.528407 (+ 51us) service_pool.cc:225] Handling call
0112 02:52:30.402242 (+2873835us) ts_tablet_manager.cc:1145] Deleting tablet 04f135b226774285a077e5806131d91b
0112 02:52:30.402252 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.402319 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2873635}
I20250112 02:52:30.402539 1609 raft_consensus.cc:513] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:30.402745 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 301) could send its response
I20250112 02:52:30.402913 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 302) took 2834 ms. Trace:
I20250112 02:52:30.403051 1175 rpcz_store.cc:276] 0112 02:52:27.568369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.568428 (+ 59us) service_pool.cc:225] Handling call
0112 02:52:30.402755 (+2834327us) ts_tablet_manager.cc:1145] Deleting tablet c4a3486ef011492c8a1df8d6278adedf
0112 02:52:30.402764 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.402850 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2834099}
W20250112 02:52:30.403333 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 302) could send its response
I20250112 02:52:30.403399 1175 tablet_replica.cc:331] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:30.404155 1175 raft_consensus.cc:2238] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:30.404505 1175 raft_consensus.cc:2267] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:30.405143 1175 ts_tablet_manager.cc:1905] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:30.407184 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51456 (request call id 406) could send its response
I20250112 02:52:30.408221 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7bd96fb381f4e73a0081be3fc799c36" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:30.408710 713 leader_election.cc:304] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
W20250112 02:52:30.409062 713 connection.cc:441] server connection from 127.0.0.1:40386 torn down before Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40386 (request call id 404) could send its response
W20250112 02:52:30.409788 714 leader_election.cc:343] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.409943 1602 raft_consensus.cc:2798] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:30.410171 1602 raft_consensus.cc:491] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:30.410282 1602 raft_consensus.cc:3054] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.410854 1173 tablet_bootstrap.cc:492] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.411000 1173 ts_tablet_manager.cc:1397] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.005s
I20250112 02:52:30.411742 1173 raft_consensus.cc:357] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.411952 1173 raft_consensus.cc:383] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.412041 1175 ts_tablet_manager.cc:1918] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:30.412055 1173 raft_consensus.cc:738] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.412153 1175 log.cc:1198] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/9163878cfeb64693a9c5493e812546af
I20250112 02:52:30.412318 1611 raft_consensus.cc:695] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:30.412257 1173 consensus_queue.cc:260] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.412513 1175 ts_tablet_manager.cc:1939] T 9163878cfeb64693a9c5493e812546af P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:30.412644 1611 consensus_queue.cc:237] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.413304 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 304) took 2778 ms. Trace:
I20250112 02:52:30.413414 1175 rpcz_store.cc:276] 0112 02:52:27.634903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.639604 (+ 4701us) service_pool.cc:225] Handling call
0112 02:52:30.403328 (+2763724us) ts_tablet_manager.cc:1145] Deleting tablet 9163878cfeb64693a9c5493e812546af
0112 02:52:30.403336 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.408803 (+ 5467us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.411929 (+ 3126us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.413267 (+ 1338us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2763387}
W20250112 02:52:30.413735 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 304) could send its response
I20250112 02:52:30.413666 1602 raft_consensus.cc:513] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.413862 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 309) took 2660 ms. Trace:
I20250112 02:52:30.413975 1175 rpcz_store.cc:276] 0112 02:52:27.753503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.753562 (+ 59us) service_pool.cc:225] Handling call
0112 02:52:30.413727 (+2660165us) ts_tablet_manager.cc:1145] Deleting tablet b61f16f3d601436bb4b32344de28c1a7
0112 02:52:30.413739 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.413828 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2659940}
W20250112 02:52:30.414177 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 309) could send its response
I20250112 02:52:30.414407 1609 leader_election.cc:290] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:30.410177 713 connection.cc:441] server connection from 127.0.0.1:40386 torn down before Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40386 (request call id 402) could send its response
I20250112 02:52:30.415030 1177 tablet_bootstrap.cc:492] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.415508 1177 ts_tablet_manager.cc:1397] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.188s user 0.037s sys 0.094s
I20250112 02:52:30.415618 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 311) took 2580 ms. Trace:
I20250112 02:52:30.416282 1177 raft_consensus.cc:357] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.416548 1177 raft_consensus.cc:383] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.416652 1177 raft_consensus.cc:738] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.415714 1175 rpcz_store.cc:276] 0112 02:52:27.835012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.835062 (+ 50us) service_pool.cc:225] Handling call
0112 02:52:30.415355 (+2580293us) ts_tablet_manager.cc:1145] Deleting tablet 87759e8b99b04e628badc8f60cb38bb3
0112 02:52:30.415364 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.415578 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2580061}
I20250112 02:52:30.417025 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd498ccce484a89855c982762086890" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:30.417112 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 310) took 2582 ms. Trace:
I20250112 02:52:30.417018 1177 consensus_queue.cc:260] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.417207 1175 rpcz_store.cc:276] 0112 02:52:27.834854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.835321 (+ 467us) service_pool.cc:225] Handling call
0112 02:52:30.416925 (+2581604us) ts_tablet_manager.cc:1145] Deleting tablet 2730634a5cfd4c2fb5c4e30100c556d4
0112 02:52:30.416933 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.417080 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2581453}
I20250112 02:52:30.417534 1177 ts_tablet_manager.cc:1428] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:30.417505 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 314) took 2497 ms. Trace:
W20250112 02:52:30.417501 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 311) could send its response
W20250112 02:52:30.417860 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 310) could send its response
I20250112 02:52:30.418066 1177 tablet_bootstrap.cc:492] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:30.418187 714 leader_election.cc:343] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.416395 1602 leader_election.cc:290] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.417665 1175 rpcz_store.cc:276] 0112 02:52:27.920032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.920257 (+ 225us) service_pool.cc:225] Handling call
0112 02:52:30.417383 (+2497126us) ts_tablet_manager.cc:1145] Deleting tablet b9c006281dc949119951c74e23167d7f
0112 02:52:30.417391 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.417462 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2496820}
I20250112 02:52:30.418784 1175 tablet_replica.cc:331] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
W20250112 02:52:30.418869 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 314) could send its response
I20250112 02:52:30.419013 1175 raft_consensus.cc:2238] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:30.419150 1175 raft_consensus.cc:2267] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:30.419941 1177 tablet_bootstrap.cc:654] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.420168 1175 ts_tablet_manager.cc:1905] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:30.420372 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7bd96fb381f4e73a0081be3fc799c36" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:30.420559 919 raft_consensus.cc:3054] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.420873 1173 ts_tablet_manager.cc:1428] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20250112 02:52:30.421180 1173 tablet_bootstrap.cc:492] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:30.422773 1173 tablet_bootstrap.cc:654] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.423117 919 raft_consensus.cc:2463] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
W20250112 02:52:30.424252 713 connection.cc:441] server connection from 127.0.0.1:40386 torn down before Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40386 (request call id 405) could send its response
W20250112 02:52:30.424384 713 connection.cc:441] server connection from 127.0.0.1:40386 torn down before Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40386 (request call id 406) could send its response
W20250112 02:52:30.424501 713 connection.cc:441] server connection from 127.0.0.1:40386 torn down before Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40386 (request call id 400) could send its response
W20250112 02:52:30.424625 713 connection.cc:441] server connection from 127.0.0.1:40386 torn down before Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40386 (request call id 403) could send its response
I20250112 02:52:30.425068 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd498ccce484a89855c982762086890" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:30.425302 919 raft_consensus.cc:3054] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.428097 1533 raft_consensus.cc:491] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.428236 1533 raft_consensus.cc:513] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.428910 1533 leader_election.cc:290] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:30.429438 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e166b3f8414ababbd4a3f07fb90c4c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:30.430202 855 leader_election.cc:343] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.430536 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e166b3f8414ababbd4a3f07fb90c4c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:30.431020 854 leader_election.cc:343] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.431169 854 leader_election.cc:304] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.431620 1533 raft_consensus.cc:2743] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.432618 713 leader_election.cc:304] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.432670 919 raft_consensus.cc:2463] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:30.432896 1177 tablet_bootstrap.cc:492] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.432917 1175 ts_tablet_manager.cc:1918] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:30.433043 1177 ts_tablet_manager.cc:1397] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20250112 02:52:30.433091 1175 log.cc:1198] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/4c0c2fad4f3243238641aae56b305bb5
I20250112 02:52:30.433265 1611 raft_consensus.cc:2798] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:30.433434 1611 raft_consensus.cc:695] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:30.433467 1175 ts_tablet_manager.cc:1939] T 4c0c2fad4f3243238641aae56b305bb5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:30.433923 1181 tablet_bootstrap.cc:492] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:30.434058 1181 ts_tablet_manager.cc:1397] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.049s user 0.012s sys 0.002s
I20250112 02:52:30.434348 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 315) took 2509 ms. Trace:
I20250112 02:52:30.434473 1175 rpcz_store.cc:276] 0112 02:52:27.924381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.929823 (+ 5442us) service_pool.cc:225] Handling call
0112 02:52:30.418728 (+2488905us) ts_tablet_manager.cc:1145] Deleting tablet 4c0c2fad4f3243238641aae56b305bb5
0112 02:52:30.418736 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.424999 (+ 6263us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.432800 (+ 7801us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.434287 (+ 1487us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2488741}
W20250112 02:52:30.434854 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 315) could send its response
I20250112 02:52:30.434816 1181 raft_consensus.cc:357] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.434968 1175 tablet_replica.cc:331] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:30.434994 1181 raft_consensus.cc:383] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.434558 1611 consensus_queue.cc:237] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.435102 1181 raft_consensus.cc:738] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.435223 1175 raft_consensus.cc:2238] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:30.435422 1175 raft_consensus.cc:2267] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:30.435360 1181 consensus_queue.cc:260] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.435914 1181 ts_tablet_manager.cc:1428] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:30.436048 1175 ts_tablet_manager.cc:1905] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:30.437084 713 leader_election.cc:304] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.437328 1602 raft_consensus.cc:2798] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:30.437467 1602 raft_consensus.cc:695] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:30.438297 1602 consensus_queue.cc:237] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.440268 1177 raft_consensus.cc:357] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.440418 1173 tablet_bootstrap.cc:492] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.440557 1173 ts_tablet_manager.cc:1397] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.010s
I20250112 02:52:30.441339 1173 raft_consensus.cc:357] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.441579 1173 raft_consensus.cc:383] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.441673 1173 raft_consensus.cc:738] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.441951 1173 consensus_queue.cc:260] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.442405 1177 raft_consensus.cc:383] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.442507 1177 raft_consensus.cc:738] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.442703 1177 consensus_queue.cc:260] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.442817 1173 ts_tablet_manager.cc:1428] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:30.444151 1173 tablet_bootstrap.cc:492] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:30.444798 1181 tablet_bootstrap.cc:492] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:30.442035 1175 ts_tablet_manager.cc:1918] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:30.445075 1175 log.cc:1198] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/3fd498ccce484a89855c982762086890
I20250112 02:52:30.445432 1175 ts_tablet_manager.cc:1939] T 3fd498ccce484a89855c982762086890 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:30.445528 1177 ts_tablet_manager.cc:1428] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250112 02:52:30.445828 1177 tablet_bootstrap.cc:492] T 65f22a7b1b804661af3b7e785a833739 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:30.446107 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 317) took 2454 ms. Trace:
I20250112 02:52:30.446213 1175 rpcz_store.cc:276] 0112 02:52:27.991512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:27.991595 (+ 83us) service_pool.cc:225] Handling call
0112 02:52:30.434830 (+2443235us) ts_tablet_manager.cc:1145] Deleting tablet 3fd498ccce484a89855c982762086890
0112 02:52:30.434845 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.439237 (+ 4392us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.441949 (+ 2712us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.446079 (+ 4130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2442995}
W20250112 02:52:30.446484 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 317) could send its response
I20250112 02:52:30.446527 1175 tablet_replica.cc:331] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:30.447000 1175 raft_consensus.cc:2238] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:30.447166 1175 raft_consensus.cc:2267] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:30.447679 1175 ts_tablet_manager.cc:1905] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:30.447729 1177 tablet_bootstrap.cc:654] T 65f22a7b1b804661af3b7e785a833739 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.447860 1181 tablet_bootstrap.cc:654] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.450006 1602 raft_consensus.cc:491] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.450189 1602 raft_consensus.cc:513] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.450778 1602 leader_election.cc:290] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.451753 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9163878cfeb64693a9c5493e812546af" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:30.452263 714 leader_election.cc:343] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.452913 1173 tablet_bootstrap.cc:654] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.455206 1181 tablet_bootstrap.cc:492] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:30.455401 1181 ts_tablet_manager.cc:1397] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20250112 02:52:30.456143 1181 raft_consensus.cc:357] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.456410 1181 raft_consensus.cc:383] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.456529 1181 raft_consensus.cc:738] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.456835 1181 consensus_queue.cc:260] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.458441 1181 ts_tablet_manager.cc:1428] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 02:52:30.459553 1181 tablet_bootstrap.cc:492] T f8c01b80fc8242d98fdfbf767212144d P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:30.460486 1175 ts_tablet_manager.cc:1918] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:30.460682 1175 log.cc:1198] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/b7bd96fb381f4e73a0081be3fc799c36
I20250112 02:52:30.461169 1175 ts_tablet_manager.cc:1939] T b7bd96fb381f4e73a0081be3fc799c36 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:30.462087 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 316) took 2470 ms. Trace:
I20250112 02:52:30.462200 1175 rpcz_store.cc:276] 0112 02:52:27.991306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:28.002654 (+ 11348us) service_pool.cc:225] Handling call
0112 02:52:30.446464 (+2443810us) ts_tablet_manager.cc:1145] Deleting tablet b7bd96fb381f4e73a0081be3fc799c36
0112 02:52:30.446478 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.456500 (+ 10022us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.460347 (+ 3847us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.462027 (+ 1680us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2443592}
I20250112 02:52:30.462580 1175 tablet_replica.cc:331] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:30.462810 1175 raft_consensus.cc:2238] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:30.462985 1175 raft_consensus.cc:2267] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:30.467574 1602 raft_consensus.cc:491] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.467711 1602 raft_consensus.cc:513] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.468241 1602 leader_election.cc:290] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.469159 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a88463991294bd8a335337f8b072c3d" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:30.469678 714 leader_election.cc:343] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:30.473122 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 316) could send its response
I20250112 02:52:30.473286 1175 ts_tablet_manager.cc:1905] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:30.475517 1181 tablet_bootstrap.cc:654] T f8c01b80fc8242d98fdfbf767212144d P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.475812 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a88463991294bd8a335337f8b072c3d" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:30.476017 916 raft_consensus.cc:2463] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
W20250112 02:52:30.476518 713 leader_election.cc:343] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 9163878cfeb64693a9c5493e812546af
I20250112 02:52:30.476717 713 leader_election.cc:304] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:30.477145 713 leader_election.cc:304] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.477278 1611 raft_consensus.cc:2743] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.477459 1177 tablet_bootstrap.cc:492] T 65f22a7b1b804661af3b7e785a833739 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.477686 1177 ts_tablet_manager.cc:1397] T 65f22a7b1b804661af3b7e785a833739 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.019s
I20250112 02:52:30.478332 1177 raft_consensus.cc:357] T 65f22a7b1b804661af3b7e785a833739 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.478657 1177 raft_consensus.cc:383] T 65f22a7b1b804661af3b7e785a833739 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.478744 1177 raft_consensus.cc:738] T 65f22a7b1b804661af3b7e785a833739 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.478562 1611 raft_consensus.cc:2798] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:30.478951 1611 raft_consensus.cc:491] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:30.479063 1611 raft_consensus.cc:3054] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.479454 1175 ts_tablet_manager.cc:1918] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:30.479585 1175 log.cc:1198] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/35f59617b34046568dd83bbb7cdc42d2
I20250112 02:52:30.479753 1177 consensus_queue.cc:260] T 65f22a7b1b804661af3b7e785a833739 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.479959 1175 ts_tablet_manager.cc:1939] T 35f59617b34046568dd83bbb7cdc42d2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:30.480618 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 331) took 2211 ms. Trace:
I20250112 02:52:30.480731 1175 rpcz_store.cc:276] 0112 02:52:28.269446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:28.269503 (+ 57us) service_pool.cc:225] Handling call
0112 02:52:30.462492 (+2192989us) ts_tablet_manager.cc:1145] Deleting tablet 35f59617b34046568dd83bbb7cdc42d2
0112 02:52:30.462503 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.476770 (+ 14267us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.479358 (+ 2588us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.480577 (+ 1219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2192791}
I20250112 02:52:30.481104 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 339) took 1968 ms. Trace:
I20250112 02:52:30.481206 1175 rpcz_store.cc:276] 0112 02:52:28.512564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:28.512628 (+ 64us) service_pool.cc:225] Handling call
0112 02:52:30.480983 (+1968355us) ts_tablet_manager.cc:1145] Deleting tablet 416fb6820cd54e118950c68d381f51b2
0112 02:52:30.480991 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.481077 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1955315}
I20250112 02:52:30.481509 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 340) took 1966 ms. Trace:
I20250112 02:52:30.481622 1175 rpcz_store.cc:276] 0112 02:52:28.515351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:28.515420 (+ 69us) service_pool.cc:225] Handling call
0112 02:52:30.481386 (+1965966us) ts_tablet_manager.cc:1145] Deleting tablet 7f4e4f1e451b48dc9cdd780300f6888c
0112 02:52:30.481395 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.481482 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1950426}
I20250112 02:52:30.481885 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 342) took 1954 ms. Trace:
I20250112 02:52:30.481969 1175 rpcz_store.cc:276] 0112 02:52:28.527697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:28.527750 (+ 53us) service_pool.cc:225] Handling call
0112 02:52:30.481790 (+1954040us) ts_tablet_manager.cc:1145] Deleting tablet a12c5d58e6c246c983e8a533906ce77d
0112 02:52:30.481797 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.481856 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1938772}
I20250112 02:52:30.482236 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 344) took 1939 ms. Trace:
I20250112 02:52:30.482329 1175 rpcz_store.cc:276] 0112 02:52:28.542672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:28.542724 (+ 52us) service_pool.cc:225] Handling call
0112 02:52:30.482129 (+1939405us) ts_tablet_manager.cc:1145] Deleting tablet f234256b343f447dadfc3d1936a824ee
0112 02:52:30.482140 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.482206 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1936018}
W20250112 02:52:30.482455 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 331) could send its response
I20250112 02:52:30.482620 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 343) took 1951 ms. Trace:
I20250112 02:52:30.482734 1175 rpcz_store.cc:276] 0112 02:52:28.531232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:28.532089 (+ 857us) service_pool.cc:225] Handling call
0112 02:52:30.482497 (+1950408us) ts_tablet_manager.cc:1145] Deleting tablet d4abf59d96414e5faa7cf75ba133832c
0112 02:52:30.482504 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.482567 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1924381}
I20250112 02:52:30.482978 1175 tablet_replica.cc:331] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:30.483160 1175 raft_consensus.cc:2238] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:30.483319 1175 raft_consensus.cc:2267] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:30.482635 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 339) could send its response
I20250112 02:52:30.483858 1175 ts_tablet_manager.cc:1905] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:30.483924 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 340) could send its response
W20250112 02:52:30.484066 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 342) could send its response
W20250112 02:52:30.484251 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 344) could send its response
W20250112 02:52:30.484393 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 343) could send its response
I20250112 02:52:30.485863 1177 ts_tablet_manager.cc:1428] T 65f22a7b1b804661af3b7e785a833739 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250112 02:52:30.489569 1175 ts_tablet_manager.cc:1918] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:30.489710 1175 log.cc:1198] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/2e9533ea72f6420bb55a53cdfc9125f7
I20250112 02:52:30.490095 1175 ts_tablet_manager.cc:1939] T 2e9533ea72f6420bb55a53cdfc9125f7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:30.490602 1177 tablet_bootstrap.cc:492] T aadb2ab1d2024f889841ac91e065e064 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:30.490727 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 346) took 1929 ms. Trace:
I20250112 02:52:30.490844 1175 rpcz_store.cc:276] 0112 02:52:28.561453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:28.562305 (+ 852us) service_pool.cc:225] Handling call
0112 02:52:30.482928 (+1920623us) ts_tablet_manager.cc:1145] Deleting tablet 2e9533ea72f6420bb55a53cdfc9125f7
0112 02:52:30.482935 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.486698 (+ 3763us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.489488 (+ 2790us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.490674 (+ 1186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1920276}
W20250112 02:52:30.491113 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 346) could send its response
I20250112 02:52:30.491207 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 345) took 1929 ms. Trace:
I20250112 02:52:30.491274 1175 rpcz_store.cc:276] 0112 02:52:28.561310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:28.563056 (+ 1746us) service_pool.cc:225] Handling call
0112 02:52:30.491086 (+1928030us) ts_tablet_manager.cc:1145] Deleting tablet 73136be25c79491784d3ea71aa9b27f2
0112 02:52:30.491093 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.491161 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1923358}
W20250112 02:52:30.491478 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 345) could send its response
I20250112 02:52:30.491597 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 347) took 1873 ms. Trace:
I20250112 02:52:30.491684 1175 rpcz_store.cc:276] 0112 02:52:28.618374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:28.618437 (+ 63us) service_pool.cc:225] Handling call
0112 02:52:30.491450 (+1873013us) ts_tablet_manager.cc:1145] Deleting tablet bb217afc08304c0ea35dee3120347ba1
0112 02:52:30.491457 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.491547 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1872788}
W20250112 02:52:30.491900 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 347) could send its response
I20250112 02:52:30.491943 1175 tablet_replica.cc:331] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:30.492244 1175 raft_consensus.cc:2238] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:30.492385 1175 raft_consensus.cc:2267] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:30.492794 1175 ts_tablet_manager.cc:1905] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:30.492961 1177 tablet_bootstrap.cc:654] T aadb2ab1d2024f889841ac91e065e064 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.494330 1602 raft_consensus.cc:491] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.494464 1602 raft_consensus.cc:513] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.495018 1602 leader_election.cc:290] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.495657 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0aba0139cf479b80b32f3dfc936535" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:30.495962 1181 tablet_bootstrap.cc:492] T f8c01b80fc8242d98fdfbf767212144d P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:30.496086 1181 ts_tablet_manager.cc:1397] T f8c01b80fc8242d98fdfbf767212144d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
W20250112 02:52:30.496155 714 leader_election.cc:343] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.496728 1181 raft_consensus.cc:357] T f8c01b80fc8242d98fdfbf767212144d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.496903 1181 raft_consensus.cc:383] T f8c01b80fc8242d98fdfbf767212144d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.496995 1181 raft_consensus.cc:738] T f8c01b80fc8242d98fdfbf767212144d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.497196 1181 consensus_queue.cc:260] T f8c01b80fc8242d98fdfbf767212144d P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.497745 1181 ts_tablet_manager.cc:1428] T f8c01b80fc8242d98fdfbf767212144d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:30.497984 1181 tablet_bootstrap.cc:492] T e3ee5d8a17874d7ba54de51568e7d2e6 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:30.499529 1181 tablet_bootstrap.cc:654] T e3ee5d8a17874d7ba54de51568e7d2e6 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.502945 1611 raft_consensus.cc:513] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.504437 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a88463991294bd8a335337f8b072c3d" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:30.504974 714 leader_election.cc:343] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:30.503702 713 leader_election.cc:343] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: fb0aba0139cf479b80b32f3dfc936535
I20250112 02:52:30.505228 713 leader_election.cc:304] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:30.505513 1602 raft_consensus.cc:2743] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.506001 1611 leader_election.cc:290] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.506489 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a88463991294bd8a335337f8b072c3d" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:30.506692 916 raft_consensus.cc:3054] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.508085 1173 tablet_bootstrap.cc:492] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.508224 1173 ts_tablet_manager.cc:1397] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.064s user 0.001s sys 0.010s
I20250112 02:52:30.508831 1181 tablet_bootstrap.cc:492] T e3ee5d8a17874d7ba54de51568e7d2e6 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:30.508965 1181 ts_tablet_manager.cc:1397] T e3ee5d8a17874d7ba54de51568e7d2e6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20250112 02:52:30.508922 1173 raft_consensus.cc:357] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.509097 1173 raft_consensus.cc:383] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.509186 1173 raft_consensus.cc:738] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.509516 916 raft_consensus.cc:2463] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:30.509426 1173 consensus_queue.cc:260] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.509553 1181 raft_consensus.cc:357] T e3ee5d8a17874d7ba54de51568e7d2e6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.509999 713 leader_election.cc:304] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.510496 1611 raft_consensus.cc:2798] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:30.509747 1181 raft_consensus.cc:383] T e3ee5d8a17874d7ba54de51568e7d2e6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.510713 1181 raft_consensus.cc:738] T e3ee5d8a17874d7ba54de51568e7d2e6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.510928 1181 consensus_queue.cc:260] T e3ee5d8a17874d7ba54de51568e7d2e6 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.511399 1181 ts_tablet_manager.cc:1428] T e3ee5d8a17874d7ba54de51568e7d2e6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:30.511900 1533 raft_consensus.cc:491] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.512046 1533 raft_consensus.cc:513] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.512548 1181 tablet_bootstrap.cc:492] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:30.512660 1533 leader_election.cc:290] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:30.513525 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc62dcbf67e84e36ae1fbe127ccd4879" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:30.513713 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc62dcbf67e84e36ae1fbe127ccd4879" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:30.514240 1181 tablet_bootstrap.cc:654] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.514387 1173 ts_tablet_manager.cc:1428] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250112 02:52:30.514719 1173 tablet_bootstrap.cc:492] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:30.514219 854 leader_election.cc:343] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:30.514932 855 leader_election.cc:343] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.515051 855 leader_election.cc:304] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.515249 1533 raft_consensus.cc:2743] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.516206 1175 ts_tablet_manager.cc:1918] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:30.516321 1175 log.cc:1198] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/65f22a7b1b804661af3b7e785a833739
I20250112 02:52:30.516647 1175 ts_tablet_manager.cc:1939] T 65f22a7b1b804661af3b7e785a833739 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:30.520068 1611 raft_consensus.cc:695] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:30.520395 1611 consensus_queue.cc:237] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.522260 1177 tablet_bootstrap.cc:492] T aadb2ab1d2024f889841ac91e065e064 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.522414 1177 ts_tablet_manager.cc:1397] T aadb2ab1d2024f889841ac91e065e064 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20250112 02:52:30.523460 1177 raft_consensus.cc:357] T aadb2ab1d2024f889841ac91e065e064 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.523711 1177 raft_consensus.cc:383] T aadb2ab1d2024f889841ac91e065e064 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.523824 1177 raft_consensus.cc:738] T aadb2ab1d2024f889841ac91e065e064 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.524065 1177 consensus_queue.cc:260] T aadb2ab1d2024f889841ac91e065e064 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.524856 1173 tablet_bootstrap.cc:654] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.525574 1177 ts_tablet_manager.cc:1428] T aadb2ab1d2024f889841ac91e065e064 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250112 02:52:30.525840 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 348) took 1788 ms. Trace:
I20250112 02:52:30.525983 1175 rpcz_store.cc:276] 0112 02:52:28.737149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:28.737208 (+ 59us) service_pool.cc:225] Handling call
0112 02:52:30.491865 (+1754657us) ts_tablet_manager.cc:1145] Deleting tablet 65f22a7b1b804661af3b7e785a833739
0112 02:52:30.491874 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.495305 (+ 3431us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.516101 (+ 20796us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.525793 (+ 9692us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1754425}
W20250112 02:52:30.526294 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 348) could send its response
I20250112 02:52:30.526530 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 349) took 1782 ms. Trace:
I20250112 02:52:30.526674 1175 rpcz_store.cc:276] 0112 02:52:28.744536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:28.747563 (+ 3027us) service_pool.cc:225] Handling call
0112 02:52:30.526303 (+1778740us) ts_tablet_manager.cc:1145] Deleting tablet 70715db0394d46d9b6b289160b00be7d
0112 02:52:30.526329 (+ 26us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.526491 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1778551}
I20250112 02:52:30.527012 1177 tablet_bootstrap.cc:492] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:30.526975 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 349) could send its response
I20250112 02:52:30.527032 1175 tablet_replica.cc:331] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:30.527312 1175 raft_consensus.cc:2238] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:30.527499 1175 raft_consensus.cc:2267] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:30.528095 1175 ts_tablet_manager.cc:1905] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:30.529000 1177 tablet_bootstrap.cc:654] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.532212 1177 tablet_bootstrap.cc:492] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.532891 1177 ts_tablet_manager.cc:1397] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250112 02:52:30.533804 1175 ts_tablet_manager.cc:1918] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:30.533926 1175 log.cc:1198] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/42cb7e7a41634cf1bdb978ac9d2befcd
I20250112 02:52:30.534235 1175 ts_tablet_manager.cc:1939] T 42cb7e7a41634cf1bdb978ac9d2befcd P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:30.534845 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 356) took 1756 ms. Trace:
I20250112 02:52:30.534930 1175 rpcz_store.cc:276] 0112 02:52:28.778283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:28.778332 (+ 49us) service_pool.cc:225] Handling call
0112 02:52:30.526962 (+1748630us) ts_tablet_manager.cc:1145] Deleting tablet 42cb7e7a41634cf1bdb978ac9d2befcd
0112 02:52:30.526972 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.531036 (+ 4064us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.533723 (+ 2687us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.534799 (+ 1076us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1748423}
I20250112 02:52:30.535183 1175 tablet_replica.cc:331] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:30.535321 1175 raft_consensus.cc:2238] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:30.535689 1175 raft_consensus.cc:2267] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:30.535912 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 356) could send its response
I20250112 02:52:30.536279 1175 ts_tablet_manager.cc:1905] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:30.536793 1602 raft_consensus.cc:491] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.536931 1602 raft_consensus.cc:513] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.537562 1602 leader_election.cc:290] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.540053 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0c2fad4f3243238641aae56b305bb5" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:30.540441 1181 tablet_bootstrap.cc:492] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:30.540585 1181 ts_tablet_manager.cc:1397] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.002s
W20250112 02:52:30.540856 714 leader_election.cc:343] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.541366 1181 raft_consensus.cc:357] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.541584 1181 raft_consensus.cc:383] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.541666 1181 raft_consensus.cc:738] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.541849 1181 consensus_queue.cc:260] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.542331 1181 ts_tablet_manager.cc:1428] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:30.542940 1181 tablet_bootstrap.cc:492] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:30.544584 1181 tablet_bootstrap.cc:654] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:30.548852 713 leader_election.cc:343] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 4c0c2fad4f3243238641aae56b305bb5
I20250112 02:52:30.549089 713 leader_election.cc:304] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:30.549417 1602 raft_consensus.cc:2743] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.550668 1177 raft_consensus.cc:357] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.551052 1177 raft_consensus.cc:383] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.551177 1177 raft_consensus.cc:738] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.551812 1177 consensus_queue.cc:260] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.552461 1175 ts_tablet_manager.cc:1918] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:30.552579 1175 log.cc:1198] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/5601a9398c9a466582a98967c0e749e2
I20250112 02:52:30.555177 1177 ts_tablet_manager.cc:1428] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20250112 02:52:30.555282 1175 ts_tablet_manager.cc:1939] T 5601a9398c9a466582a98967c0e749e2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:30.555929 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 359) took 1735 ms. Trace:
I20250112 02:52:30.556025 1175 rpcz_store.cc:276] 0112 02:52:28.820949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:28.821003 (+ 54us) service_pool.cc:225] Handling call
0112 02:52:30.535117 (+1714114us) ts_tablet_manager.cc:1145] Deleting tablet 5601a9398c9a466582a98967c0e749e2
0112 02:52:30.535125 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.540278 (+ 5153us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.552360 (+ 12082us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.555895 (+ 3535us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1712833}
W20250112 02:52:30.556269 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 359) could send its response
I20250112 02:52:30.556598 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 361) took 1728 ms. Trace:
I20250112 02:52:30.556703 1175 rpcz_store.cc:276] 0112 02:52:28.828602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:28.828651 (+ 49us) service_pool.cc:225] Handling call
0112 02:52:30.556480 (+1727829us) ts_tablet_manager.cc:1145] Deleting tablet b1d9fae290ec4281839991f9d40b105c
0112 02:52:30.556488 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.556562 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1727642}
W20250112 02:52:30.556891 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 361) could send its response
I20250112 02:52:30.557178 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 362) took 1674 ms. Trace:
I20250112 02:52:30.557277 1175 rpcz_store.cc:276] 0112 02:52:28.882429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:28.882478 (+ 49us) service_pool.cc:225] Handling call
0112 02:52:30.557070 (+1674592us) ts_tablet_manager.cc:1145] Deleting tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b
0112 02:52:30.557077 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.557142 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1674382}
W20250112 02:52:30.557507 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 362) could send its response
I20250112 02:52:30.557775 1175 tablet_replica.cc:331] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:30.557965 1175 raft_consensus.cc:2238] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:30.558112 1175 raft_consensus.cc:2267] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:30.558616 1175 ts_tablet_manager.cc:1905] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:30.558835 1173 tablet_bootstrap.cc:492] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.558995 1177 tablet_bootstrap.cc:492] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:30.559038 1173 ts_tablet_manager.cc:1397] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.044s user 0.014s sys 0.006s
I20250112 02:52:30.559752 1173 raft_consensus.cc:357] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.559979 1181 tablet_bootstrap.cc:492] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:30.560071 1173 raft_consensus.cc:383] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.560154 1181 ts_tablet_manager.cc:1397] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250112 02:52:30.560199 1173 raft_consensus.cc:738] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.560572 1177 tablet_bootstrap.cc:654] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.560556 1173 consensus_queue.cc:260] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.560914 1181 raft_consensus.cc:357] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.561120 1181 raft_consensus.cc:383] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.561192 1181 raft_consensus.cc:738] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.561391 1181 consensus_queue.cc:260] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.561892 1181 ts_tablet_manager.cc:1428] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:30.562251 1181 tablet_bootstrap.cc:492] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:30.563872 1181 tablet_bootstrap.cc:654] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.565816 1173 ts_tablet_manager.cc:1428] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250112 02:52:30.571172 1175 ts_tablet_manager.cc:1918] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:30.571307 1175 log.cc:1198] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/aadb2ab1d2024f889841ac91e065e064
I20250112 02:52:30.571722 1175 ts_tablet_manager.cc:1939] T aadb2ab1d2024f889841ac91e065e064 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:30.572736 1177 tablet_bootstrap.cc:492] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.572948 1177 ts_tablet_manager.cc:1397] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.002s
I20250112 02:52:30.573760 1177 raft_consensus.cc:357] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.574038 1177 raft_consensus.cc:383] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.574164 1177 raft_consensus.cc:738] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.574656 1177 consensus_queue.cc:260] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.578246 1181 tablet_bootstrap.cc:492] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:30.578401 1181 ts_tablet_manager.cc:1397] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20250112 02:52:30.578511 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 365) took 1593 ms. Trace:
I20250112 02:52:30.578615 1175 rpcz_store.cc:276] 0112 02:52:28.985413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:28.990605 (+ 5192us) service_pool.cc:225] Handling call
0112 02:52:30.557711 (+1567106us) ts_tablet_manager.cc:1145] Deleting tablet aadb2ab1d2024f889841ac91e065e064
0112 02:52:30.557718 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.567549 (+ 9831us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.571056 (+ 3507us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.578451 (+ 7395us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1566911}
W20250112 02:52:30.578858 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 365) could send its response
I20250112 02:52:30.579241 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 366) took 1547 ms. Trace:
I20250112 02:52:30.579213 1181 raft_consensus.cc:357] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.579420 1181 raft_consensus.cc:383] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.579331 1175 rpcz_store.cc:276] 0112 02:52:29.031635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:29.031687 (+ 52us) service_pool.cc:225] Handling call
0112 02:52:30.579050 (+1547363us) ts_tablet_manager.cc:1145] Deleting tablet 8c821ea3ca4946c290c17135366e0cac
0112 02:52:30.579059 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.579172 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1547138}
I20250112 02:52:30.579548 1181 raft_consensus.cc:738] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
W20250112 02:52:30.579607 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 366) could send its response
I20250112 02:52:30.579756 1173 tablet_bootstrap.cc:492] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:30.579775 1181 consensus_queue.cc:260] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.580253 1177 ts_tablet_manager.cc:1428] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250112 02:52:30.580297 1181 ts_tablet_manager.cc:1428] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:30.580579 1181 tablet_bootstrap.cc:492] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:30.580659 1177 tablet_bootstrap.cc:492] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:30.581637 1173 tablet_bootstrap.cc:654] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.582226 1181 tablet_bootstrap.cc:654] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.582384 1177 tablet_bootstrap.cc:654] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.579885 1175 tablet_replica.cc:331] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:30.583459 1175 raft_consensus.cc:2238] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:30.583726 1175 raft_consensus.cc:2267] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:30.584270 1175 ts_tablet_manager.cc:1905] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:30.591881 1175 ts_tablet_manager.cc:1918] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:30.591997 1175 log.cc:1198] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/b6dc79f4d1194f81963a8094ed6b0d2e
I20250112 02:52:30.592430 1175 ts_tablet_manager.cc:1939] T b6dc79f4d1194f81963a8094ed6b0d2e P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:30.592856 1173 tablet_bootstrap.cc:492] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.592980 1173 ts_tablet_manager.cc:1397] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20250112 02:52:30.593127 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 367) took 1553 ms. Trace:
I20250112 02:52:30.593226 1175 rpcz_store.cc:276] 0112 02:52:29.039579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:29.039628 (+ 49us) service_pool.cc:225] Handling call
0112 02:52:30.579807 (+1540179us) ts_tablet_manager.cc:1145] Deleting tablet b6dc79f4d1194f81963a8094ed6b0d2e
0112 02:52:30.579815 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.587915 (+ 8100us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.591770 (+ 3855us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.593093 (+ 1323us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1539885}
I20250112 02:52:30.593541 1175 tablet_replica.cc:331] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
W20250112 02:52:30.593660 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 367) could send its response
I20250112 02:52:30.593750 1175 raft_consensus.cc:2238] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:30.593884 1175 raft_consensus.cc:2267] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:30.594350 1175 ts_tablet_manager.cc:1905] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:30.594350 1173 raft_consensus.cc:357] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.594552 1173 raft_consensus.cc:383] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.594669 1173 raft_consensus.cc:738] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.594933 1173 consensus_queue.cc:260] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.595781 1177 tablet_bootstrap.cc:492] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.595963 1177 ts_tablet_manager.cc:1397] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20250112 02:52:30.596820 1177 raft_consensus.cc:357] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.597055 1177 raft_consensus.cc:383] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.597138 1177 raft_consensus.cc:738] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.597389 1177 consensus_queue.cc:260] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.600337 1175 ts_tablet_manager.cc:1918] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:30.600483 1175 log.cc:1198] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/3c686fdc8896463c81c9f7c5f53a3da2
I20250112 02:52:30.600812 1175 ts_tablet_manager.cc:1939] T 3c686fdc8896463c81c9f7c5f53a3da2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:30.601171 1177 ts_tablet_manager.cc:1428] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250112 02:52:30.601517 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 368) took 1547 ms. Trace:
I20250112 02:52:30.601615 1175 rpcz_store.cc:276] 0112 02:52:29.053909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:29.053960 (+ 51us) service_pool.cc:225] Handling call
0112 02:52:30.593488 (+1539528us) ts_tablet_manager.cc:1145] Deleting tablet 3c686fdc8896463c81c9f7c5f53a3da2
0112 02:52:30.593496 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.597336 (+ 3840us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.600243 (+ 2907us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.601458 (+ 1215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1539338}
W20250112 02:52:30.601929 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 368) could send its response
I20250112 02:52:30.602080 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 369) took 1535 ms. Trace:
I20250112 02:52:30.602178 1175 rpcz_store.cc:276] 0112 02:52:29.066469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:29.066531 (+ 62us) service_pool.cc:225] Handling call
0112 02:52:30.601910 (+1535379us) ts_tablet_manager.cc:1145] Deleting tablet daef3a58605e40978f7389698b22265d
0112 02:52:30.601921 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.602025 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1535159}
W20250112 02:52:30.602406 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 369) could send its response
I20250112 02:52:30.602591 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 370) took 1524 ms. Trace:
I20250112 02:52:30.602685 1175 rpcz_store.cc:276] 0112 02:52:29.078132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:29.078199 (+ 67us) service_pool.cc:225] Handling call
0112 02:52:30.602422 (+1524223us) ts_tablet_manager.cc:1145] Deleting tablet fe614eaeb793473d9b2da206b7345562
0112 02:52:30.602432 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.602540 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1524041}
W20250112 02:52:30.602908 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 370) could send its response
I20250112 02:52:30.603027 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 371) took 1515 ms. Trace:
I20250112 02:52:30.603152 1175 rpcz_store.cc:276] 0112 02:52:29.087212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:29.087453 (+ 241us) service_pool.cc:225] Handling call
0112 02:52:30.602902 (+1515449us) ts_tablet_manager.cc:1145] Deleting tablet 06edfca60eed4dd3a2dee7b662b0970c
0112 02:52:30.602910 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.602973 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1515201}
I20250112 02:52:30.603446 1177 tablet_bootstrap.cc:492] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:30.603446 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 371) could send its response
I20250112 02:52:30.604257 1173 ts_tablet_manager.cc:1428] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250112 02:52:30.605525 1173 tablet_bootstrap.cc:492] T 27ab58e45d0d44b283019240f44b66bf P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:30.606040 1177 tablet_bootstrap.cc:654] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.607208 1173 tablet_bootstrap.cc:654] T 27ab58e45d0d44b283019240f44b66bf P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.603528 1175 tablet_replica.cc:331] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:30.608208 1175 raft_consensus.cc:2238] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:30.608429 1175 raft_consensus.cc:2267] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:30.608945 1175 ts_tablet_manager.cc:1905] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:30.609511 1181 tablet_bootstrap.cc:492] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:30.609789 1181 ts_tablet_manager.cc:1397] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.001s
I20250112 02:52:30.610818 1181 raft_consensus.cc:357] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.611181 1181 raft_consensus.cc:383] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.611318 1181 raft_consensus.cc:738] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.611728 1181 consensus_queue.cc:260] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.612398 1181 ts_tablet_manager.cc:1428] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 02:52:30.615182 1177 tablet_bootstrap.cc:492] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.615226 1175 ts_tablet_manager.cc:1918] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:30.615347 1177 ts_tablet_manager.cc:1397] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20250112 02:52:30.615387 1175 log.cc:1198] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/b9199b0d2b924a79938a979df4196bb4
I20250112 02:52:30.615514 1181 tablet_bootstrap.cc:492] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:30.615818 1175 ts_tablet_manager.cc:1939] T b9199b0d2b924a79938a979df4196bb4 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:30.616149 1177 raft_consensus.cc:357] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.616338 1177 raft_consensus.cc:383] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.616430 1177 raft_consensus.cc:738] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.616518 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 372) took 1401 ms. Trace:
I20250112 02:52:30.616626 1175 rpcz_store.cc:276] 0112 02:52:29.214657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:29.214708 (+ 51us) service_pool.cc:225] Handling call
0112 02:52:30.603424 (+1388716us) ts_tablet_manager.cc:1145] Deleting tablet b9199b0d2b924a79938a979df4196bb4
0112 02:52:30.603444 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.612185 (+ 8741us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.615099 (+ 2914us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.616484 (+ 1385us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1388509}
I20250112 02:52:30.616976 1175 tablet_replica.cc:331] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
W20250112 02:52:30.616978 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 372) could send its response
I20250112 02:52:30.617238 1175 raft_consensus.cc:2238] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:30.617419 1173 tablet_bootstrap.cc:492] T 27ab58e45d0d44b283019240f44b66bf P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.617568 1173 ts_tablet_manager.cc:1397] T 27ab58e45d0d44b283019240f44b66bf P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250112 02:52:30.617590 1175 raft_consensus.cc:2267] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:30.618120 1175 ts_tablet_manager.cc:1905] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:30.618178 1181 tablet_bootstrap.cc:654] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.618516 1537 raft_consensus.cc:491] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.618655 1537 raft_consensus.cc:513] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.619287 1537 leader_election.cc:290] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.618294 1173 raft_consensus.cc:357] T 27ab58e45d0d44b283019240f44b66bf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.619799 1173 raft_consensus.cc:383] T 27ab58e45d0d44b283019240f44b66bf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.619889 1173 raft_consensus.cc:738] T 27ab58e45d0d44b283019240f44b66bf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.620148 1173 consensus_queue.cc:260] T 27ab58e45d0d44b283019240f44b66bf P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:30.621119 991 leader_election.cc:343] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: aba1744138fe42ae9d097db01cac083d
W20250112 02:52:30.622992 989 leader_election.cc:343] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: aba1744138fe42ae9d097db01cac083d
I20250112 02:52:30.623127 1173 ts_tablet_manager.cc:1428] T 27ab58e45d0d44b283019240f44b66bf P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 02:52:30.623195 989 leader_election.cc:304] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:30.624258 1173 tablet_bootstrap.cc:492] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:30.625128 1175 ts_tablet_manager.cc:1918] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:30.616652 1177 consensus_queue.cc:260] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.625247 1175 log.cc:1198] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/96e201d798c94e448a78e1c8adf41724
I20250112 02:52:30.625658 1175 ts_tablet_manager.cc:1939] T 96e201d798c94e448a78e1c8adf41724 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:30.625677 1177 ts_tablet_manager.cc:1428] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250112 02:52:30.626606 1177 tablet_bootstrap.cc:492] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:30.627590 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 392) took 1275 ms. Trace:
I20250112 02:52:30.627688 1175 rpcz_store.cc:276] 0112 02:52:29.351971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:29.358153 (+ 6182us) service_pool.cc:225] Handling call
0112 02:52:30.616889 (+1258736us) ts_tablet_manager.cc:1145] Deleting tablet 96e201d798c94e448a78e1c8adf41724
0112 02:52:30.616896 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.622171 (+ 5275us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.625036 (+ 2865us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:30.626311 (+ 1275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1258474}
I20250112 02:52:30.627878 1173 tablet_bootstrap.cc:654] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:30.627992 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 392) could send its response
I20250112 02:52:30.628185 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 391) took 1276 ms. Trace:
I20250112 02:52:30.628268 1175 rpcz_store.cc:276] 0112 02:52:29.351895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:29.358453 (+ 6558us) service_pool.cc:225] Handling call
0112 02:52:30.627997 (+1269544us) ts_tablet_manager.cc:1145] Deleting tablet d45fce86e02f46d4a57af718fa2a62c8
0112 02:52:30.628006 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.628132 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1269313}
W20250112 02:52:30.628523 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 391) could send its response
I20250112 02:52:30.628656 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 390) took 1276 ms. Trace:
I20250112 02:52:30.628744 1175 rpcz_store.cc:276] 0112 02:52:29.351819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:29.358708 (+ 6889us) service_pool.cc:225] Handling call
0112 02:52:30.628516 (+1269808us) ts_tablet_manager.cc:1145] Deleting tablet d659c5a4f7a446c6aaf2283f65b45106
0112 02:52:30.628525 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.628603 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1269597}
W20250112 02:52:30.628935 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 390) could send its response
I20250112 02:52:30.629078 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 389) took 1277 ms. Trace:
I20250112 02:52:30.629165 1175 rpcz_store.cc:276] 0112 02:52:29.351723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:29.358976 (+ 7253us) service_pool.cc:225] Handling call
0112 02:52:30.628947 (+1269971us) ts_tablet_manager.cc:1145] Deleting tablet f776e2a6ec704cf9b64483bc7d5b1dff
0112 02:52:30.628956 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.629026 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1269763}
W20250112 02:52:30.629379 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 389) could send its response
I20250112 02:52:30.629501 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 396) took 1197 ms. Trace:
I20250112 02:52:30.629591 1175 rpcz_store.cc:276] 0112 02:52:29.431861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:29.431910 (+ 49us) service_pool.cc:225] Handling call
0112 02:52:30.629359 (+1197449us) ts_tablet_manager.cc:1145] Deleting tablet b6d84027ad6d40a5be5afcdd854431a5
0112 02:52:30.629379 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.629448 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1197267}
W20250112 02:52:30.629810 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 396) could send its response
I20250112 02:52:30.629972 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 398) took 1191 ms. Trace:
I20250112 02:52:30.630055 1175 rpcz_store.cc:276] 0112 02:52:29.438964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:29.439009 (+ 45us) service_pool.cc:225] Handling call
0112 02:52:30.629822 (+1190813us) ts_tablet_manager.cc:1145] Deleting tablet 02d0f67141dc46b6a87ca0ad080d5bfe
0112 02:52:30.629829 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.629919 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1190643}
I20250112 02:52:30.630271 1177 tablet_bootstrap.cc:654] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:30.630295 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 398) could send its response
I20250112 02:52:30.630435 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 397) took 1196 ms. Trace:
I20250112 02:52:30.630532 1175 rpcz_store.cc:276] 0112 02:52:29.433859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:29.433906 (+ 47us) service_pool.cc:225] Handling call
0112 02:52:30.630286 (+1196380us) ts_tablet_manager.cc:1145] Deleting tablet 352c87f1dc724e7b94bc2e64abfce8d7
0112 02:52:30.630295 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:30.630379 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1180134}
W20250112 02:52:30.630748 853 connection.cc:441] server connection from 127.0.0.1:51456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51456 (request call id 397) could send its response
I20250112 02:52:30.632308 1173 tablet_bootstrap.cc:492] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.632542 1173 ts_tablet_manager.cc:1397] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20250112 02:52:30.633507 1173 raft_consensus.cc:357] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.634151 1173 raft_consensus.cc:383] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.634477 1173 raft_consensus.cc:738] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.634747 1173 consensus_queue.cc:260] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.635330 1173 ts_tablet_manager.cc:1428] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 02:52:30.635370 1537 raft_consensus.cc:2743] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.636809 1173 tablet_bootstrap.cc:492] T ab2e31a65294407bb62b13e92898a7d2 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:30.637548 1533 raft_consensus.cc:491] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.637761 1533 raft_consensus.cc:513] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.638548 1173 tablet_bootstrap.cc:654] T ab2e31a65294407bb62b13e92898a7d2 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.638633 1533 leader_election.cc:290] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:30.639560 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4916ae3917084cfe8c93240a8857be0e" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:30.640313 855 leader_election.cc:343] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.643329 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4916ae3917084cfe8c93240a8857be0e" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:30.644286 854 leader_election.cc:343] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.644460 854 leader_election.cc:304] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.646548 1533 raft_consensus.cc:2743] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.647652 1177 tablet_bootstrap.cc:492] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.647859 1177 ts_tablet_manager.cc:1397] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.002s
I20250112 02:52:30.648870 1177 raft_consensus.cc:357] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.649164 1177 raft_consensus.cc:383] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.649240 1177 raft_consensus.cc:738] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.649511 1177 consensus_queue.cc:260] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.653067 1177 ts_tablet_manager.cc:1428] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 02:52:30.654340 1177 tablet_bootstrap.cc:492] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:30.656749 1173 tablet_bootstrap.cc:492] T ab2e31a65294407bb62b13e92898a7d2 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.656934 1173 ts_tablet_manager.cc:1397] T ab2e31a65294407bb62b13e92898a7d2 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.007s
I20250112 02:52:30.657728 1177 tablet_bootstrap.cc:654] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.657649 1173 raft_consensus.cc:357] T ab2e31a65294407bb62b13e92898a7d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.657790 1533 raft_consensus.cc:491] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.657896 1173 raft_consensus.cc:383] T ab2e31a65294407bb62b13e92898a7d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.657974 1173 raft_consensus.cc:738] T ab2e31a65294407bb62b13e92898a7d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.657927 1533 raft_consensus.cc:513] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.658195 1173 consensus_queue.cc:260] T ab2e31a65294407bb62b13e92898a7d2 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.658514 1533 leader_election.cc:290] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:30.658528 991 consensus_peers.cc:487] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b521522c4784d2ab4056268823f1099. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:30.659363 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f135b226774285a077e5806131d91b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:30.658803 1173 ts_tablet_manager.cc:1428] T ab2e31a65294407bb62b13e92898a7d2 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250112 02:52:30.659931 855 leader_election.cc:343] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.660164 1173 tablet_bootstrap.cc:492] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:30.660409 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f135b226774285a077e5806131d91b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:30.660981 854 leader_election.cc:343] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.661121 854 leader_election.cc:304] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.661499 1533 raft_consensus.cc:2743] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.661870 1173 tablet_bootstrap.cc:654] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.662498 1181 tablet_bootstrap.cc:492] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:30.662649 1181 ts_tablet_manager.cc:1397] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.028s
I20250112 02:52:30.662969 1177 tablet_bootstrap.cc:492] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.663059 1177 ts_tablet_manager.cc:1397] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20250112 02:52:30.664945 1181 raft_consensus.cc:357] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.665177 1181 raft_consensus.cc:383] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.665270 1181 raft_consensus.cc:738] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.665262 1177 raft_consensus.cc:357] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.665460 1177 raft_consensus.cc:383] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.665565 1177 raft_consensus.cc:738] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.665797 1177 consensus_queue.cc:260] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.667740 1181 consensus_queue.cc:260] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.668309 1181 ts_tablet_manager.cc:1428] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250112 02:52:30.668646 1181 tablet_bootstrap.cc:492] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:30.670338 1181 tablet_bootstrap.cc:654] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.671203 1173 tablet_bootstrap.cc:492] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.671360 1173 ts_tablet_manager.cc:1397] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20250112 02:52:30.670087 1177 ts_tablet_manager.cc:1428] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250112 02:52:30.673381 1611 raft_consensus.cc:491] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.673518 1611 raft_consensus.cc:513] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.674100 1611 leader_election.cc:290] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:30.674746 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8eb0efc5d7f4c72bdfc7531a444d42a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:30.672155 1173 raft_consensus.cc:357] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.674916 1173 raft_consensus.cc:383] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.674955 916 raft_consensus.cc:2463] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:30.675058 1173 raft_consensus.cc:738] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.675246 1181 tablet_bootstrap.cc:492] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:30.675321 1173 consensus_queue.cc:260] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.675417 1181 ts_tablet_manager.cc:1397] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20250112 02:52:30.675639 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8eb0efc5d7f4c72bdfc7531a444d42a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:30.675822 1173 ts_tablet_manager.cc:1428] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250112 02:52:30.676095 1173 tablet_bootstrap.cc:492] T d8eb0efc5d7f4c72bdfc7531a444d42a P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:30.676441 713 leader_election.cc:304] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
W20250112 02:52:30.676854 714 leader_election.cc:343] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.677354 1611 raft_consensus.cc:2798] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:30.677459 1611 raft_consensus.cc:491] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:30.677548 1611 raft_consensus.cc:3054] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.678128 1173 tablet_bootstrap.cc:654] T d8eb0efc5d7f4c72bdfc7531a444d42a P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.678433 1181 raft_consensus.cc:357] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.678630 1181 raft_consensus.cc:383] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.678716 1181 raft_consensus.cc:738] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.678942 1181 consensus_queue.cc:260] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.679436 1181 ts_tablet_manager.cc:1428] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20250112 02:52:30.679803 1181 tablet_bootstrap.cc:492] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:30.680292 1611 raft_consensus.cc:513] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.682538 1181 tablet_bootstrap.cc:654] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.682642 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8eb0efc5d7f4c72bdfc7531a444d42a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:30.682864 916 raft_consensus.cc:3054] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.684635 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8eb0efc5d7f4c72bdfc7531a444d42a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:30.688544 1533 raft_consensus.cc:491] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 02:52:30.688738 714 leader_election.cc:343] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.688724 1533 raft_consensus.cc:513] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.689144 1177 tablet_bootstrap.cc:492] T 4916ae3917084cfe8c93240a8857be0e P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:30.689345 1533 leader_election.cc:290] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:30.689846 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161fe26b9c0e4543b7aeb0021506d72a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:30.690477 854 leader_election.cc:343] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.690536 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161fe26b9c0e4543b7aeb0021506d72a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:30.691002 855 leader_election.cc:343] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.691126 855 leader_election.cc:304] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.691198 916 raft_consensus.cc:2463] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:30.691589 713 leader_election.cc:304] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.691768 1533 raft_consensus.cc:2743] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.691890 1602 raft_consensus.cc:2798] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:30.692090 1602 raft_consensus.cc:695] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:30.692409 1602 consensus_queue.cc:237] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.692772 1177 tablet_bootstrap.cc:654] T 4916ae3917084cfe8c93240a8857be0e P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.701731 1177 tablet_bootstrap.cc:492] T 4916ae3917084cfe8c93240a8857be0e P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.701916 1177 ts_tablet_manager.cc:1397] T 4916ae3917084cfe8c93240a8857be0e P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20250112 02:52:30.702932 1611 leader_election.cc:290] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:30.702840 1177 raft_consensus.cc:357] T 4916ae3917084cfe8c93240a8857be0e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.703171 1177 raft_consensus.cc:383] T 4916ae3917084cfe8c93240a8857be0e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.703258 1177 raft_consensus.cc:738] T 4916ae3917084cfe8c93240a8857be0e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.703553 1177 consensus_queue.cc:260] T 4916ae3917084cfe8c93240a8857be0e P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.704270 1177 ts_tablet_manager.cc:1428] T 4916ae3917084cfe8c93240a8857be0e P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250112 02:52:30.704594 1177 tablet_bootstrap.cc:492] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:30.706679 1177 tablet_bootstrap.cc:654] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.708137 1689 raft_consensus.cc:491] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f4a49a5756b143fcb767c3f41463f8d9)
I20250112 02:52:30.712471 1689 raft_consensus.cc:513] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
W20250112 02:52:30.718261 439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20250112 02:52:30.718495 1689 leader_election.cc:290] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f4a49a5756b143fcb767c3f41463f8d9 (127.0.104.61:38771), dd25f03f25df49178a5c53c636ecf304 (127.0.104.60:39987)
I20250112 02:52:30.720412 1602 raft_consensus.cc:491] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.720587 1602 raft_consensus.cc:513] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.721671 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f59617b34046568dd83bbb7cdc42d2" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:30.722282 714 leader_election.cc:343] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.723079 1602 leader_election.cc:290] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:30.723874 713 leader_election.cc:343] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 35f59617b34046568dd83bbb7cdc42d2
I20250112 02:52:30.724035 713 leader_election.cc:304] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:30.724361 1177 tablet_bootstrap.cc:492] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.724498 1177 ts_tablet_manager.cc:1397] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20250112 02:52:30.724540 1602 raft_consensus.cc:2743] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.725328 1177 raft_consensus.cc:357] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.725517 1177 raft_consensus.cc:383] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.725595 1177 raft_consensus.cc:738] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.725795 1177 consensus_queue.cc:260] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.726339 1177 ts_tablet_manager.cc:1428] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 02:52:30.728672 1533 raft_consensus.cc:491] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.728843 1533 raft_consensus.cc:513] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.729643 1533 leader_election.cc:290] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:30.730643 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015412a5fa304453bd8d1be1c3cd7a46" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:30.731590 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015412a5fa304453bd8d1be1c3cd7a46" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:30.732438 854 leader_election.cc:343] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:30.732817 855 leader_election.cc:343] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.732959 855 leader_election.cc:304] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.733529 1533 raft_consensus.cc:2743] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.733623 1181 tablet_bootstrap.cc:492] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:30.733752 1181 ts_tablet_manager.cc:1397] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.054s user 0.029s sys 0.011s
I20250112 02:52:30.734477 1181 raft_consensus.cc:357] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.734647 1181 raft_consensus.cc:383] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.734726 1181 raft_consensus.cc:738] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.735014 1177 tablet_bootstrap.cc:492] T 64707ee432fb4e96a3bbf7252dfa1813 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:30.734938 1181 consensus_queue.cc:260] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.737068 1177 tablet_bootstrap.cc:654] T 64707ee432fb4e96a3bbf7252dfa1813 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.735440 1181 ts_tablet_manager.cc:1428] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250112 02:52:30.738135 439 leader_election.cc:336] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f4a49a5756b143fcb767c3f41463f8d9 (127.0.104.61:38771): Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection refused (error 111)
I20250112 02:52:30.738358 1181 tablet_bootstrap.cc:492] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:30.738799 1173 tablet_bootstrap.cc:492] T d8eb0efc5d7f4c72bdfc7531a444d42a P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.738958 1173 ts_tablet_manager.cc:1397] T d8eb0efc5d7f4c72bdfc7531a444d42a P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.063s user 0.020s sys 0.030s
I20250112 02:52:30.739974 1181 tablet_bootstrap.cc:654] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.739955 1173 raft_consensus.cc:357] T d8eb0efc5d7f4c72bdfc7531a444d42a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.740176 1173 raft_consensus.cc:383] T d8eb0efc5d7f4c72bdfc7531a444d42a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.740263 1173 raft_consensus.cc:738] T d8eb0efc5d7f4c72bdfc7531a444d42a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.740521 1173 consensus_queue.cc:260] T d8eb0efc5d7f4c72bdfc7531a444d42a P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.741091 1173 ts_tablet_manager.cc:1428] T d8eb0efc5d7f4c72bdfc7531a444d42a P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:30.743235 1173 tablet_bootstrap.cc:492] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:30.744097 1602 raft_consensus.cc:491] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.744113 1634 raft_consensus.cc:491] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.744218 1602 raft_consensus.cc:513] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.744258 1634 raft_consensus.cc:513] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.744844 1634 leader_election.cc:290] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.745187 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c686fdc8896463c81c9f7c5f53a3da2" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:30.745504 1177 tablet_bootstrap.cc:492] T 64707ee432fb4e96a3bbf7252dfa1813 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.745570 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9163878cfeb64693a9c5493e812546af" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:30.745649 1177 ts_tablet_manager.cc:1397] T 64707ee432fb4e96a3bbf7252dfa1813 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20250112 02:52:30.745779 779 raft_consensus.cc:2463] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:30.746271 991 leader_election.cc:304] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:30.746325 1177 raft_consensus.cc:357] T 64707ee432fb4e96a3bbf7252dfa1813 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.746507 1177 raft_consensus.cc:383] T 64707ee432fb4e96a3bbf7252dfa1813 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.746578 1177 raft_consensus.cc:738] T 64707ee432fb4e96a3bbf7252dfa1813 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.746635 1634 raft_consensus.cc:2798] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:30.746742 1634 raft_consensus.cc:491] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:30.746824 1634 raft_consensus.cc:3054] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.746835 1177 consensus_queue.cc:260] T 64707ee432fb4e96a3bbf7252dfa1813 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:30.746909 714 leader_election.cc:343] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:30.747071 989 leader_election.cc:343] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 9163878cfeb64693a9c5493e812546af
I20250112 02:52:30.748601 1602 leader_election.cc:290] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:30.749362 713 leader_election.cc:343] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 3c686fdc8896463c81c9f7c5f53a3da2
I20250112 02:52:30.749472 713 leader_election.cc:304] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:30.745191 1173 tablet_bootstrap.cc:654] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.749823 1634 raft_consensus.cc:513] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.750483 1634 leader_election.cc:290] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.751145 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9163878cfeb64693a9c5493e812546af" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:30.751327 779 raft_consensus.cc:3054] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.753939 1602 raft_consensus.cc:2743] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:30.754022 989 leader_election.cc:343] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 9163878cfeb64693a9c5493e812546af
I20250112 02:52:30.754051 779 raft_consensus.cc:2463] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:30.754139 1177 ts_tablet_manager.cc:1428] T 64707ee432fb4e96a3bbf7252dfa1813 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250112 02:52:30.754392 991 leader_election.cc:304] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:30.755002 1634 raft_consensus.cc:2798] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:30.755564 1394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "436220c23fa948efb5b6ca15875d8b0e" candidate_term: 4 candidate_status { last_received { term: 3 index: 514 } } ignore_live_leader: false dest_uuid: "dd25f03f25df49178a5c53c636ecf304" is_pre_election: true
I20250112 02:52:30.755870 1394 raft_consensus.cc:2463] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 436220c23fa948efb5b6ca15875d8b0e in term 3.
I20250112 02:52:30.756098 1181 tablet_bootstrap.cc:492] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:30.756253 1181 ts_tablet_manager.cc:1397] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250112 02:52:30.756354 441 leader_election.cc:304] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 436220c23fa948efb5b6ca15875d8b0e, dd25f03f25df49178a5c53c636ecf304; no voters: f4a49a5756b143fcb767c3f41463f8d9
I20250112 02:52:30.756623 1689 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250112 02:52:30.756747 1689 raft_consensus.cc:491] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 3 FOLLOWER]: Starting leader election (detected failure of leader f4a49a5756b143fcb767c3f41463f8d9)
I20250112 02:52:30.756846 1689 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 3 FOLLOWER]: Advancing to term 4
I20250112 02:52:30.756958 1602 raft_consensus.cc:491] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.757076 1602 raft_consensus.cc:513] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.758033 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ab58e45d0d44b283019240f44b66bf" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:30.758208 916 raft_consensus.cc:2463] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:30.758553 1602 leader_election.cc:290] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.758534 713 leader_election.cc:304] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:30.759369 1602 raft_consensus.cc:2798] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:30.759590 1602 raft_consensus.cc:491] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:30.759694 1602 raft_consensus.cc:3054] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.760584 1689 raft_consensus.cc:513] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:30.761495 1602 raft_consensus.cc:513] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.762005 1689 leader_election.cc:290] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [CANDIDATE]: Term 4 election: Requested vote from peers f4a49a5756b143fcb767c3f41463f8d9 (127.0.104.61:38771), dd25f03f25df49178a5c53c636ecf304 (127.0.104.60:39987)
W20250112 02:52:30.762457 439 leader_election.cc:336] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer f4a49a5756b143fcb767c3f41463f8d9 (127.0.104.61:38771): Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection refused (error 111)
I20250112 02:52:30.762471 1394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "436220c23fa948efb5b6ca15875d8b0e" candidate_term: 4 candidate_status { last_received { term: 3 index: 514 } } ignore_live_leader: false dest_uuid: "dd25f03f25df49178a5c53c636ecf304"
I20250112 02:52:30.762624 1394 raft_consensus.cc:3054] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 3 FOLLOWER]: Advancing to term 4
I20250112 02:52:30.762665 1181 raft_consensus.cc:357] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.762920 1181 raft_consensus.cc:383] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.763016 1181 raft_consensus.cc:738] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.763311 1181 consensus_queue.cc:260] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.764034 1181 ts_tablet_manager.cc:1428] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250112 02:52:30.764652 1602 leader_election.cc:290] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.765058 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ab58e45d0d44b283019240f44b66bf" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:30.765959 1177 tablet_bootstrap.cc:492] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:30.766734 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ab58e45d0d44b283019240f44b66bf" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:30.766908 916 raft_consensus.cc:3054] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.768257 1177 tablet_bootstrap.cc:654] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.769440 916 raft_consensus.cc:2463] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:30.769630 1181 tablet_bootstrap.cc:492] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:30.770148 1394 raft_consensus.cc:2463] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 436220c23fa948efb5b6ca15875d8b0e in term 4.
I20250112 02:52:30.770174 713 leader_election.cc:304] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:30.770816 441 leader_election.cc:304] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 436220c23fa948efb5b6ca15875d8b0e, dd25f03f25df49178a5c53c636ecf304; no voters: f4a49a5756b143fcb767c3f41463f8d9
I20250112 02:52:30.771880 1181 tablet_bootstrap.cc:654] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.772168 1689 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 4 FOLLOWER]: Leader election won for term 4
I20250112 02:52:30.772397 1602 raft_consensus.cc:2798] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:30.772948 1602 raft_consensus.cc:695] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:30.773212 1602 consensus_queue.cc:237] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.774921 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ab58e45d0d44b283019240f44b66bf" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:30.775322 1051 raft_consensus.cc:3054] T 27ab58e45d0d44b283019240f44b66bf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.775656 1173 tablet_bootstrap.cc:492] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.775775 1173 ts_tablet_manager.cc:1397] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.033s user 0.014s sys 0.007s
I20250112 02:52:30.776513 1173 raft_consensus.cc:357] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.776705 1173 raft_consensus.cc:383] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.776786 1173 raft_consensus.cc:738] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.776990 1173 consensus_queue.cc:260] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.777503 1173 ts_tablet_manager.cc:1428] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:30.777776 1173 tablet_bootstrap.cc:492] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:30.778004 1051 raft_consensus.cc:2463] T 27ab58e45d0d44b283019240f44b66bf P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:30.778873 1634 raft_consensus.cc:695] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:30.779095 1177 tablet_bootstrap.cc:492] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.779227 1177 ts_tablet_manager.cc:1397] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250112 02:52:30.780122 1173 tablet_bootstrap.cc:654] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.780028 1177 raft_consensus.cc:357] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.780223 1177 raft_consensus.cc:383] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.780313 1177 raft_consensus.cc:738] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.780529 1177 consensus_queue.cc:260] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.780617 1634 consensus_queue.cc:237] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.781195 1177 ts_tablet_manager.cc:1428] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:30.781502 1177 tablet_bootstrap.cc:492] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:30.783221 1177 tablet_bootstrap.cc:654] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.785122 1533 raft_consensus.cc:491] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.785284 1533 raft_consensus.cc:513] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.785979 1689 raft_consensus.cc:695] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [term 4 LEADER]: Becoming Leader. State: Replica: 436220c23fa948efb5b6ca15875d8b0e, State: Running, Role: LEADER
I20250112 02:52:30.785987 1533 leader_election.cc:290] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:30.786706 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039a4734358646508093200d59d30013" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:30.786737 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039a4734358646508093200d59d30013" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:30.787184 855 leader_election.cc:343] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:30.787297 854 leader_election.cc:343] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.787379 855 leader_election.cc:304] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.787810 1533 raft_consensus.cc:2743] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.787698 1689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 513, Committed index: 513, Last appended: 3.514, Last appended by leader: 514, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:30.788719 1181 tablet_bootstrap.cc:492] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:30.788885 1181 ts_tablet_manager.cc:1397] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.003s
I20250112 02:52:30.789597 1181 raft_consensus.cc:357] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:30.789520 1700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 436220c23fa948efb5b6ca15875d8b0e. Latest consensus state: current_term: 4 leader_uuid: "436220c23fa948efb5b6ca15875d8b0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:30.789829 1181 raft_consensus.cc:383] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.789947 1181 raft_consensus.cc:738] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.789970 1700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [sys.catalog]: This master's current role is: LEADER
I20250112 02:52:30.790176 1181 consensus_queue.cc:260] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:30.790848 1181 ts_tablet_manager.cc:1428] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:30.791375 1181 tablet_bootstrap.cc:492] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:30.792951 1181 tablet_bootstrap.cc:654] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.796367 1177 tablet_bootstrap.cc:492] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.796499 1177 ts_tablet_manager.cc:1397] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20250112 02:52:30.797175 1177 raft_consensus.cc:357] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.797366 1177 raft_consensus.cc:383] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.797428 1177 raft_consensus.cc:738] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.797681 1177 consensus_queue.cc:260] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.799700 1173 tablet_bootstrap.cc:492] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.800081 1173 ts_tablet_manager.cc:1397] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.005s
I20250112 02:52:30.800921 1177 ts_tablet_manager.cc:1428] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 02:52:30.801303 1177 tablet_bootstrap.cc:492] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:30.800835 1173 raft_consensus.cc:357] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.802408 1173 raft_consensus.cc:383] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.802507 1173 raft_consensus.cc:738] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.802759 1173 consensus_queue.cc:260] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.803287 1173 ts_tablet_manager.cc:1428] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:30.804040 1173 tablet_bootstrap.cc:492] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:30.804435 1177 tablet_bootstrap.cc:654] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.805835 1181 tablet_bootstrap.cc:492] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:30.805971 1181 ts_tablet_manager.cc:1397] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20250112 02:52:30.806666 1173 tablet_bootstrap.cc:654] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.806815 1181 raft_consensus.cc:357] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.807013 1181 raft_consensus.cc:383] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.807090 1181 raft_consensus.cc:738] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.807323 1181 consensus_queue.cc:260] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.809269 1181 ts_tablet_manager.cc:1428] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:30.809535 1181 tablet_bootstrap.cc:492] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:30.810750 1177 tablet_bootstrap.cc:492] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.810920 1177 ts_tablet_manager.cc:1397] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20250112 02:52:30.811643 1181 tablet_bootstrap.cc:654] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.811719 1173 tablet_bootstrap.cc:492] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.811828 1173 ts_tablet_manager.cc:1397] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20250112 02:52:30.812420 1173 raft_consensus.cc:357] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.812852 1173 raft_consensus.cc:383] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.812963 1173 raft_consensus.cc:738] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.813176 1173 consensus_queue.cc:260] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.814236 1602 raft_consensus.cc:491] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.814383 1602 raft_consensus.cc:513] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.814910 1602 leader_election.cc:290] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.814981 1177 raft_consensus.cc:357] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.815147 1177 raft_consensus.cc:383] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.815235 1177 raft_consensus.cc:738] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.815483 1177 consensus_queue.cc:260] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.815586 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e166b3f8414ababbd4a3f07fb90c4c" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:30.815940 1177 ts_tablet_manager.cc:1428] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
W20250112 02:52:30.816231 714 leader_election.cc:343] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.816776 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e166b3f8414ababbd4a3f07fb90c4c" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:30.816938 916 raft_consensus.cc:2463] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:30.817137 1177 tablet_bootstrap.cc:492] T 7cd9985cd20148f195a857c478b0755f P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:30.818645 1173 ts_tablet_manager.cc:1428] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:30.819311 1177 tablet_bootstrap.cc:654] T 7cd9985cd20148f195a857c478b0755f P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.819497 713 leader_election.cc:304] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.819834 1611 raft_consensus.cc:2798] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:30.819950 1611 raft_consensus.cc:491] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:30.820057 1611 raft_consensus.cc:3054] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.823215 1611 raft_consensus.cc:513] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.823846 1533 raft_consensus.cc:491] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.824004 1173 tablet_bootstrap.cc:492] T b9199b0d2b924a79938a979df4196bb4 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:30.823966 1533 raft_consensus.cc:513] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.824549 1533 leader_election.cc:290] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:30.824852 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c03b5e612d64314b210c11686fdfa2a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:30.825330 854 leader_election.cc:343] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.825459 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c03b5e612d64314b210c11686fdfa2a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:30.825829 1173 tablet_bootstrap.cc:654] T b9199b0d2b924a79938a979df4196bb4 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:30.826032 855 leader_election.cc:343] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.826153 855 leader_election.cc:304] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.826750 1533 raft_consensus.cc:2743] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.829018 1533 raft_consensus.cc:491] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.829142 1533 raft_consensus.cc:513] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.829777 1533 leader_election.cc:290] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:30.830181 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23e7a2e88c374e949ced92c557225cbf" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:30.830343 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23e7a2e88c374e949ced92c557225cbf" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:30.830513 779 raft_consensus.cc:2463] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
W20250112 02:52:30.830710 989 consensus_peers.cc:487] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b521522c4784d2ab4056268823f1099. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:52:30.830734 854 leader_election.cc:343] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.830878 855 leader_election.cc:304] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.831251 1533 raft_consensus.cc:2798] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:30.831355 1533 raft_consensus.cc:491] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:30.831437 1177 tablet_bootstrap.cc:492] T 7cd9985cd20148f195a857c478b0755f P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.831445 1533 raft_consensus.cc:3054] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.831601 1177 ts_tablet_manager.cc:1397] T 7cd9985cd20148f195a857c478b0755f P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20250112 02:52:30.832212 1177 raft_consensus.cc:357] T 7cd9985cd20148f195a857c478b0755f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.832404 1177 raft_consensus.cc:383] T 7cd9985cd20148f195a857c478b0755f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.832477 1177 raft_consensus.cc:738] T 7cd9985cd20148f195a857c478b0755f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.832741 1177 consensus_queue.cc:260] T 7cd9985cd20148f195a857c478b0755f P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.833319 1177 ts_tablet_manager.cc:1428] T 7cd9985cd20148f195a857c478b0755f P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:30.833860 1602 raft_consensus.cc:491] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.833925 1177 tablet_bootstrap.cc:492] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:30.833966 1533 raft_consensus.cc:513] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.833981 1602 raft_consensus.cc:513] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.834483 1602 leader_election.cc:290] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.834790 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23e7a2e88c374e949ced92c557225cbf" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:30.834957 779 raft_consensus.cc:3054] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.835573 1177 tablet_bootstrap.cc:654] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:30.836274 713 leader_election.cc:343] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: b9199b0d2b924a79938a979df4196bb4
I20250112 02:52:30.837082 1173 tablet_bootstrap.cc:492] T b9199b0d2b924a79938a979df4196bb4 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.837208 1173 ts_tablet_manager.cc:1397] T b9199b0d2b924a79938a979df4196bb4 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.001s
I20250112 02:52:30.837481 779 raft_consensus.cc:2463] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:30.837829 1173 raft_consensus.cc:357] T b9199b0d2b924a79938a979df4196bb4 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.838021 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9199b0d2b924a79938a979df4196bb4" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:30.838176 855 leader_election.cc:304] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:30.838042 1173 raft_consensus.cc:383] T b9199b0d2b924a79938a979df4196bb4 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.838352 1173 raft_consensus.cc:738] T b9199b0d2b924a79938a979df4196bb4 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.838577 1173 consensus_queue.cc:260] T b9199b0d2b924a79938a979df4196bb4 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.838760 1714 raft_consensus.cc:2798] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:30.838814 1051 raft_consensus.cc:2463] T b9199b0d2b924a79938a979df4196bb4 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:30.839182 714 leader_election.cc:304] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:30.839641 1602 raft_consensus.cc:2798] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:30.839751 1602 raft_consensus.cc:491] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:30.839838 1602 raft_consensus.cc:3054] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.840019 1611 leader_election.cc:290] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.840497 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e166b3f8414ababbd4a3f07fb90c4c" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:30.840729 916 raft_consensus.cc:3054] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.840947 1533 leader_election.cc:290] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:30.843102 1602 raft_consensus.cc:513] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.843533 1181 tablet_bootstrap.cc:492] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:30.843663 1181 ts_tablet_manager.cc:1397] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.008s
I20250112 02:52:30.844156 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e166b3f8414ababbd4a3f07fb90c4c" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:30.844424 1181 raft_consensus.cc:357] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:30.844650 1181 raft_consensus.cc:383] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 02:52:30.844599 714 leader_election.cc:343] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.844729 1181 raft_consensus.cc:738] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.844718 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23e7a2e88c374e949ced92c557225cbf" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:30.844964 1181 consensus_queue.cc:260] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:30.845438 1602 leader_election.cc:290] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:30.845505 854 leader_election.cc:343] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.845655 1181 ts_tablet_manager.cc:1428] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:30.846320 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9199b0d2b924a79938a979df4196bb4" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:30.846537 1051 raft_consensus.cc:3054] T b9199b0d2b924a79938a979df4196bb4 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.845911 1181 tablet_bootstrap.cc:492] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:30.848321 1634 raft_consensus.cc:491] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.848610 1181 tablet_bootstrap.cc:654] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.848489 1634 raft_consensus.cc:513] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.849061 1634 leader_election.cc:290] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.849859 1051 raft_consensus.cc:2463] T b9199b0d2b924a79938a979df4196bb4 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
W20250112 02:52:30.850755 713 leader_election.cc:343] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: b9199b0d2b924a79938a979df4196bb4
I20250112 02:52:30.852092 916 raft_consensus.cc:2463] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:30.852627 713 leader_election.cc:304] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.852636 714 leader_election.cc:304] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:30.853489 1611 raft_consensus.cc:2798] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:30.853646 1611 raft_consensus.cc:695] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:30.853871 1611 consensus_queue.cc:237] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:30.854301 989 leader_election.cc:343] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: afa2f8aabe5e42ccbc15b193fdc03563
I20250112 02:52:30.854764 1609 raft_consensus.cc:2798] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:30.846535 1173 ts_tablet_manager.cc:1428] T b9199b0d2b924a79938a979df4196bb4 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250112 02:52:30.855417 1173 tablet_bootstrap.cc:492] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:30.856757 991 leader_election.cc:343] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: afa2f8aabe5e42ccbc15b193fdc03563
I20250112 02:52:30.856916 991 leader_election.cc:304] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:30.857188 1173 tablet_bootstrap.cc:654] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.857339 1634 raft_consensus.cc:2743] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.860185 1634 raft_consensus.cc:491] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.860370 1634 raft_consensus.cc:513] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.861107 1634 leader_election.cc:290] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.861907 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d06e2e8c13464db8cf801a292bdc99" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:30.862207 916 raft_consensus.cc:2388] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1e775296eb746db8b318f17d26820e6 in current term 1: Already voted for candidate 289afc116aa74408aa5bff74e11c50d8 in this term.
I20250112 02:52:30.863915 1634 raft_consensus.cc:491] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.864046 1634 raft_consensus.cc:513] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.864682 1634 leader_election.cc:290] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.865283 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f22a7b1b804661af3b7e785a833739" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:30.865482 780 raft_consensus.cc:2463] T 65f22a7b1b804661af3b7e785a833739 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:30.865645 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d06e2e8c13464db8cf801a292bdc99" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:30.866078 991 leader_election.cc:304] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:30.866406 991 leader_election.cc:304] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:30.866729 1537 raft_consensus.cc:2798] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:30.866847 1537 raft_consensus.cc:491] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:30.866956 1537 raft_consensus.cc:3054] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:30.867095 989 leader_election.cc:343] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 65f22a7b1b804661af3b7e785a833739
I20250112 02:52:30.867411 1634 raft_consensus.cc:3054] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.870374 1537 raft_consensus.cc:513] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.870666 1634 raft_consensus.cc:2743] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.871695 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f22a7b1b804661af3b7e785a833739" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:30.871914 779 raft_consensus.cc:3054] T 65f22a7b1b804661af3b7e785a833739 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.873287 1537 leader_election.cc:290] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:30.873485 989 leader_election.cc:343] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 65f22a7b1b804661af3b7e785a833739
I20250112 02:52:30.875840 779 raft_consensus.cc:2463] T 65f22a7b1b804661af3b7e785a833739 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:30.876293 991 leader_election.cc:304] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:30.876852 1537 raft_consensus.cc:2798] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:30.877015 1537 raft_consensus.cc:695] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:30.877281 1537 consensus_queue.cc:237] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.879271 1714 raft_consensus.cc:695] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:30.879727 1714 consensus_queue.cc:237] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.881502 1609 raft_consensus.cc:695] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:30.882042 1609 consensus_queue.cc:237] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.884848 1177 tablet_bootstrap.cc:492] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:30.884986 1177 ts_tablet_manager.cc:1397] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.051s user 0.018s sys 0.020s
I20250112 02:52:30.886090 1177 raft_consensus.cc:357] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.886301 1177 raft_consensus.cc:383] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.886408 1177 raft_consensus.cc:738] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.886622 1177 consensus_queue.cc:260] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.887171 1177 ts_tablet_manager.cc:1428] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 02:52:30.889739 1173 tablet_bootstrap.cc:492] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:30.889765 1181 tablet_bootstrap.cc:492] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:30.889870 1173 ts_tablet_manager.cc:1397] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.007s
I20250112 02:52:30.890084 1181 ts_tablet_manager.cc:1397] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.000s
I20250112 02:52:30.890414 1611 consensus_queue.cc:1035] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:30.890588 1173 raft_consensus.cc:357] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.890794 1173 raft_consensus.cc:383] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.890884 1173 raft_consensus.cc:738] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.891065 1173 consensus_queue.cc:260] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.891367 1181 raft_consensus.cc:357] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:30.891585 1173 ts_tablet_manager.cc:1428] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:30.891621 1181 raft_consensus.cc:383] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:30.891891 1173 tablet_bootstrap.cc:492] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:30.892282 1181 raft_consensus.cc:738] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:30.892794 1181 consensus_queue.cc:260] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:30.893306 1181 ts_tablet_manager.cc:1428] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:30.893858 1173 tablet_bootstrap.cc:654] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.893894 1177 tablet_bootstrap.cc:492] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:30.895641 1177 tablet_bootstrap.cc:654] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.897447 1181 tablet_bootstrap.cc:492] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:30.899588 1051 raft_consensus.cc:3054] T ecfc7d8ba5e4406ebc61f37e6d500473 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.903548 1181 tablet_bootstrap.cc:654] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.903569 1634 raft_consensus.cc:491] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.903769 1634 raft_consensus.cc:513] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.905061 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c686fdc8896463c81c9f7c5f53a3da2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:30.905298 779 raft_consensus.cc:2463] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:30.905745 991 leader_election.cc:304] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:30.906687 1602 consensus_queue.cc:1035] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:30.917829 1611 raft_consensus.cc:491] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.918057 1611 raft_consensus.cc:513] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.918592 1602 consensus_queue.cc:1035] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:30.919162 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb1a197d7e40689ccad82a44492371" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:30.922026 714 leader_election.cc:343] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:30.926391 989 leader_election.cc:343] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 3c686fdc8896463c81c9f7c5f53a3da2
I20250112 02:52:30.926643 1634 leader_election.cc:290] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.926875 1634 raft_consensus.cc:2798] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:30.927125 1634 raft_consensus.cc:491] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:30.927289 1634 raft_consensus.cc:3054] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.932279 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb1a197d7e40689ccad82a44492371" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:30.932554 919 raft_consensus.cc:2463] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:30.933048 713 leader_election.cc:304] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.933614 1602 raft_consensus.cc:2798] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:30.933718 1602 raft_consensus.cc:491] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:30.933840 1602 raft_consensus.cc:3054] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.937278 1602 raft_consensus.cc:513] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.939096 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb1a197d7e40689ccad82a44492371" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:30.939831 714 leader_election.cc:343] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.940100 1537 raft_consensus.cc:491] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.940258 1537 raft_consensus.cc:513] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.938035 1611 leader_election.cc:290] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.941304 1611 raft_consensus.cc:491] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.941468 1611 raft_consensus.cc:513] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:30.941741 989 leader_election.cc:343] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 4c0c2fad4f3243238641aae56b305bb5
I20250112 02:52:30.942025 1537 leader_election.cc:290] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.942515 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cb7e7a41634cf1bdb978ac9d2befcd" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:30.943080 714 leader_election.cc:343] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.946523 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fb1a197d7e40689ccad82a44492371" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:30.946767 919 raft_consensus.cc:3054] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.947795 1602 leader_election.cc:290] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:30.948675 713 leader_election.cc:343] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 42cb7e7a41634cf1bdb978ac9d2befcd
I20250112 02:52:30.948836 713 leader_election.cc:304] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:30.949082 1602 raft_consensus.cc:2743] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.949489 1611 leader_election.cc:290] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.950534 919 raft_consensus.cc:2463] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:30.951164 713 leader_election.cc:304] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.953500 1634 raft_consensus.cc:513] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:30.955019 989 leader_election.cc:343] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 3c686fdc8896463c81c9f7c5f53a3da2
I20250112 02:52:30.955279 1634 leader_election.cc:290] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:30.956765 1602 raft_consensus.cc:2798] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:30.956960 1602 raft_consensus.cc:695] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:30.957304 1602 consensus_queue.cc:237] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.958599 1718 raft_consensus.cc:491] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.958820 1718 raft_consensus.cc:513] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.960014 1718 leader_election.cc:290] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:30.961836 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c639a2bcd4740e4be7b2b162004669a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:30.962102 919 raft_consensus.cc:2463] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:30.962833 713 leader_election.cc:304] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:30.963510 1718 consensus_queue.cc:1035] T ecfc7d8ba5e4406ebc61f37e6d500473 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:30.964318 1718 raft_consensus.cc:2798] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:30.964453 1718 raft_consensus.cc:491] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:30.964576 1718 raft_consensus.cc:3054] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.973870 1718 raft_consensus.cc:513] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:30.962177 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c639a2bcd4740e4be7b2b162004669a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:30.978093 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 72386.5us
W20250112 02:52:30.978448 714 leader_election.cc:343] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.978808 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0c2fad4f3243238641aae56b305bb5" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:30.979099 779 raft_consensus.cc:2463] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:30.979880 1715 raft_consensus.cc:491] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.980144 1715 raft_consensus.cc:513] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.981547 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfaad67d5e34c0a8338dccc0ef05466" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:30.982091 854 leader_election.cc:343] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.982372 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c686fdc8896463c81c9f7c5f53a3da2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:30.982661 779 raft_consensus.cc:3054] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.984277 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfaad67d5e34c0a8338dccc0ef05466" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:30.984946 855 leader_election.cc:343] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:30.985085 855 leader_election.cc:304] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:30.985507 1715 leader_election.cc:290] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:30.985872 1715 raft_consensus.cc:2743] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:30.986145 779 raft_consensus.cc:2463] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:30.987197 991 leader_election.cc:304] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:30.987933 991 leader_election.cc:304] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:30.988420 1728 raft_consensus.cc:2798] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:30.988569 1728 raft_consensus.cc:491] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:30.988744 1728 raft_consensus.cc:3054] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:30.992645 1729 raft_consensus.cc:2798] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:30.992893 1729 raft_consensus.cc:695] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:30.993321 1729 consensus_queue.cc:237] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:30.999017 1727 raft_consensus.cc:491] T ab2e31a65294407bb62b13e92898a7d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:30.999218 1727 raft_consensus.cc:513] T ab2e31a65294407bb62b13e92898a7d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:31.000581 1727 leader_election.cc:290] T ab2e31a65294407bb62b13e92898a7d2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.000739 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2e31a65294407bb62b13e92898a7d2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:31.001072 918 raft_consensus.cc:2388] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1e775296eb746db8b318f17d26820e6 in current term 1: Already voted for candidate 289afc116aa74408aa5bff74e11c50d8 in this term.
I20250112 02:52:31.002632 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c639a2bcd4740e4be7b2b162004669a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:31.002847 918 raft_consensus.cc:3054] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.006397 918 raft_consensus.cc:2463] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:31.006908 713 leader_election.cc:304] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:31.007645 1611 raft_consensus.cc:2798] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.007815 1611 raft_consensus.cc:695] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:31.008141 1611 consensus_queue.cc:237] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:31.010545 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2e31a65294407bb62b13e92898a7d2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.011237 991 leader_election.cc:304] T ab2e31a65294407bb62b13e92898a7d2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:31.011705 1727 raft_consensus.cc:3054] T ab2e31a65294407bb62b13e92898a7d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.015599 1727 raft_consensus.cc:2743] T ab2e31a65294407bb62b13e92898a7d2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.018425 1181 tablet_bootstrap.cc:492] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.018592 1181 ts_tablet_manager.cc:1397] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.121s user 0.012s sys 0.025s
I20250112 02:52:31.019438 1181 raft_consensus.cc:357] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.020262 1181 raft_consensus.cc:383] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.020401 1181 raft_consensus.cc:738] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.021443 1181 consensus_queue.cc:260] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.022270 1181 ts_tablet_manager.cc:1428] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250112 02:52:31.026259 1181 tablet_bootstrap.cc:492] T 9f0314aa03b1473ebbe05ce73adf7838 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.027966 1181 tablet_bootstrap.cc:654] T 9f0314aa03b1473ebbe05ce73adf7838 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:30.992287 1728 raft_consensus.cc:513] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.031579 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0c2fad4f3243238641aae56b305bb5" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:31.031836 779 raft_consensus.cc:3054] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.035194 779 raft_consensus.cc:2463] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:31.036010 1729 raft_consensus.cc:491] T d8eb0efc5d7f4c72bdfc7531a444d42a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.036517 1729 raft_consensus.cc:513] T d8eb0efc5d7f4c72bdfc7531a444d42a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:31.037794 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8eb0efc5d7f4c72bdfc7531a444d42a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:31.038130 919 raft_consensus.cc:2388] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1e775296eb746db8b318f17d26820e6 in current term 1: Already voted for candidate 289afc116aa74408aa5bff74e11c50d8 in this term.
I20250112 02:52:31.038208 1181 tablet_bootstrap.cc:492] T 9f0314aa03b1473ebbe05ce73adf7838 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.038357 1181 ts_tablet_manager.cc:1397] T 9f0314aa03b1473ebbe05ce73adf7838 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s
I20250112 02:52:31.036314 991 leader_election.cc:304] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:31.039160 1181 raft_consensus.cc:357] T 9f0314aa03b1473ebbe05ce73adf7838 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.039566 1634 raft_consensus.cc:2798] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.039822 1634 raft_consensus.cc:695] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:31.040220 1634 consensus_queue.cc:237] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.039424 1181 raft_consensus.cc:383] T 9f0314aa03b1473ebbe05ce73adf7838 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.040500 1181 raft_consensus.cc:738] T 9f0314aa03b1473ebbe05ce73adf7838 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.040730 1181 consensus_queue.cc:260] T 9f0314aa03b1473ebbe05ce73adf7838 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.042124 1181 ts_tablet_manager.cc:1428] T 9f0314aa03b1473ebbe05ce73adf7838 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:31.042404 1181 tablet_bootstrap.cc:492] T ed7243c6329d4f11aa25eea2b7b67683 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.043282 1718 leader_election.cc:290] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:31.044622 989 leader_election.cc:343] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 4c0c2fad4f3243238641aae56b305bb5
I20250112 02:52:31.047312 1181 tablet_bootstrap.cc:654] T ed7243c6329d4f11aa25eea2b7b67683 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.048079 1729 leader_election.cc:290] T d8eb0efc5d7f4c72bdfc7531a444d42a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.049932 1728 leader_election.cc:290] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.051970 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c639a2bcd4740e4be7b2b162004669a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:31.054805 1181 tablet_bootstrap.cc:492] T ed7243c6329d4f11aa25eea2b7b67683 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.054975 1181 ts_tablet_manager.cc:1397] T ed7243c6329d4f11aa25eea2b7b67683 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20250112 02:52:31.055833 1181 raft_consensus.cc:357] T ed7243c6329d4f11aa25eea2b7b67683 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.056088 1181 raft_consensus.cc:383] T ed7243c6329d4f11aa25eea2b7b67683 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.056205 1181 raft_consensus.cc:738] T ed7243c6329d4f11aa25eea2b7b67683 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.056460 1181 consensus_queue.cc:260] T ed7243c6329d4f11aa25eea2b7b67683 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.057124 1181 ts_tablet_manager.cc:1428] T ed7243c6329d4f11aa25eea2b7b67683 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:31.057528 1181 tablet_bootstrap.cc:492] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.058041 1634 raft_consensus.cc:491] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.058419 1634 raft_consensus.cc:513] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.059226 1181 tablet_bootstrap.cc:654] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.059947 1634 leader_election.cc:290] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.060246 1715 raft_consensus.cc:491] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.060478 1715 raft_consensus.cc:513] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:31.061107 991 leader_election.cc:343] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: 2fdeaf5fa8cc4a2b97b2178d8ecbad69
W20250112 02:52:31.061345 989 leader_election.cc:343] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 2fdeaf5fa8cc4a2b97b2178d8ecbad69
I20250112 02:52:31.061473 989 leader_election.cc:304] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:31.062026 1634 raft_consensus.cc:2743] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.062793 1715 leader_election.cc:290] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:31.063155 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64707ee432fb4e96a3bbf7252dfa1813" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:31.063755 854 leader_election.cc:343] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.067273 1181 tablet_bootstrap.cc:492] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.067498 1181 ts_tablet_manager.cc:1397] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20250112 02:52:31.068442 1181 raft_consensus.cc:357] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:31.068732 1181 raft_consensus.cc:383] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.068823 1181 raft_consensus.cc:738] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.069093 1181 consensus_queue.cc:260] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:31.069757 1181 ts_tablet_manager.cc:1428] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:31.070086 1181 tablet_bootstrap.cc:492] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.049273 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8eb0efc5d7f4c72bdfc7531a444d42a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:31.052616 714 leader_election.cc:343] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.062043 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64707ee432fb4e96a3bbf7252dfa1813" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.072907 780 raft_consensus.cc:2463] T 64707ee432fb4e96a3bbf7252dfa1813 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:31.073045 991 leader_election.cc:304] T d8eb0efc5d7f4c72bdfc7531a444d42a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:31.073455 855 leader_election.cc:304] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.073558 1634 raft_consensus.cc:3054] T d8eb0efc5d7f4c72bdfc7531a444d42a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.074272 1715 raft_consensus.cc:2798] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.074404 1715 raft_consensus.cc:491] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.074522 1715 raft_consensus.cc:3054] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.076522 1181 tablet_bootstrap.cc:654] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.076650 1634 raft_consensus.cc:2743] T d8eb0efc5d7f4c72bdfc7531a444d42a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.076848 1634 raft_consensus.cc:491] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.077047 1634 raft_consensus.cc:513] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:31.078457 991 leader_election.cc:343] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: cb724feccc7f4fbc846fb6fa61be3385
I20250112 02:52:31.078861 1634 leader_election.cc:290] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:31.079319 989 leader_election.cc:343] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: cb724feccc7f4fbc846fb6fa61be3385
I20250112 02:52:31.079507 989 leader_election.cc:304] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:31.080271 1177 tablet_bootstrap.cc:492] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:31.080435 1177 ts_tablet_manager.cc:1397] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.187s user 0.012s sys 0.057s
I20250112 02:52:31.080911 1634 raft_consensus.cc:2743] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.081153 1177 raft_consensus.cc:357] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.081377 1177 raft_consensus.cc:383] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.081461 1177 raft_consensus.cc:738] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.081717 1177 consensus_queue.cc:260] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.082782 1173 tablet_bootstrap.cc:492] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.082926 1173 ts_tablet_manager.cc:1397] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.191s user 0.088s sys 0.085s
I20250112 02:52:31.083199 1177 ts_tablet_manager.cc:1428] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:31.077940 1715 raft_consensus.cc:513] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.083698 1173 raft_consensus.cc:357] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.083884 1173 raft_consensus.cc:383] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.083949 1173 raft_consensus.cc:738] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.084124 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64707ee432fb4e96a3bbf7252dfa1813" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:31.084151 1173 consensus_queue.cc:260] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.084308 780 raft_consensus.cc:3054] T 64707ee432fb4e96a3bbf7252dfa1813 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.084339 1177 tablet_bootstrap.cc:492] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:31.084697 1173 ts_tablet_manager.cc:1428] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:31.085943 1177 tablet_bootstrap.cc:654] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.086972 1173 tablet_bootstrap.cc:492] T 55fb1a197d7e40689ccad82a44492371 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:31.086977 780 raft_consensus.cc:2463] T 64707ee432fb4e96a3bbf7252dfa1813 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:31.087430 855 leader_election.cc:304] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:31.088784 1714 raft_consensus.cc:2798] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.088909 1173 tablet_bootstrap.cc:654] T 55fb1a197d7e40689ccad82a44492371 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.088937 1714 raft_consensus.cc:695] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:31.089154 1714 consensus_queue.cc:237] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.099061 1715 leader_election.cc:290] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:31.103072 1181 tablet_bootstrap.cc:492] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.103243 1181 ts_tablet_manager.cc:1397] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.004s
I20250112 02:52:31.103973 1181 raft_consensus.cc:357] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.104279 1181 raft_consensus.cc:383] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.104422 1181 raft_consensus.cc:738] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.104741 1181 consensus_queue.cc:260] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.106369 1181 ts_tablet_manager.cc:1428] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:31.106768 1181 tablet_bootstrap.cc:492] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.109094 1181 tablet_bootstrap.cc:654] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.110481 1715 raft_consensus.cc:491] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.110656 1715 raft_consensus.cc:513] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.110884 1177 tablet_bootstrap.cc:492] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:31.111068 1177 ts_tablet_manager.cc:1397] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.007s
I20250112 02:52:31.111765 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a57eb366f594f93a488c8250e195014" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.111982 1177 raft_consensus.cc:357] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.112208 1177 raft_consensus.cc:383] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 02:52:31.112219 855 leader_election.cc:343] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.112298 1177 raft_consensus.cc:738] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.112461 1715 leader_election.cc:290] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:31.100971 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64707ee432fb4e96a3bbf7252dfa1813" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:31.112599 1177 consensus_queue.cc:260] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.114369 1177 ts_tablet_manager.cc:1428] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250112 02:52:31.115670 1173 tablet_bootstrap.cc:492] T 55fb1a197d7e40689ccad82a44492371 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.115855 1173 ts_tablet_manager.cc:1397] T 55fb1a197d7e40689ccad82a44492371 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.029s user 0.018s sys 0.007s
I20250112 02:52:31.116802 1173 raft_consensus.cc:357] T 55fb1a197d7e40689ccad82a44492371 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.117029 1173 raft_consensus.cc:383] T 55fb1a197d7e40689ccad82a44492371 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.117138 1173 raft_consensus.cc:738] T 55fb1a197d7e40689ccad82a44492371 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.118593 1177 tablet_bootstrap.cc:492] T 7f4e4f1e451b48dc9cdd780300f6888c P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:31.119899 854 leader_election.cc:343] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.120208 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a57eb366f594f93a488c8250e195014" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:31.120823 854 leader_election.cc:343] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.120985 1177 tablet_bootstrap.cc:654] T 7f4e4f1e451b48dc9cdd780300f6888c P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.120992 854 leader_election.cc:304] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.121448 1715 raft_consensus.cc:2743] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.122124 1173 consensus_queue.cc:260] T 55fb1a197d7e40689ccad82a44492371 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.122747 1173 ts_tablet_manager.cc:1428] T 55fb1a197d7e40689ccad82a44492371 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250112 02:52:31.123286 1173 tablet_bootstrap.cc:492] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:31.125689 1173 tablet_bootstrap.cc:654] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.127180 1177 tablet_bootstrap.cc:492] T 7f4e4f1e451b48dc9cdd780300f6888c P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:31.128918 1177 ts_tablet_manager.cc:1397] T 7f4e4f1e451b48dc9cdd780300f6888c P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20250112 02:52:31.129608 1177 raft_consensus.cc:357] T 7f4e4f1e451b48dc9cdd780300f6888c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.129890 1177 raft_consensus.cc:383] T 7f4e4f1e451b48dc9cdd780300f6888c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.130023 1177 raft_consensus.cc:738] T 7f4e4f1e451b48dc9cdd780300f6888c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.130545 1177 consensus_queue.cc:260] T 7f4e4f1e451b48dc9cdd780300f6888c P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.132714 1177 ts_tablet_manager.cc:1428] T 7f4e4f1e451b48dc9cdd780300f6888c P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 02:52:31.133021 1177 tablet_bootstrap.cc:492] T c04b64bc2f8348a086ed3c4e59f09b16 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:31.135901 1177 tablet_bootstrap.cc:654] T c04b64bc2f8348a086ed3c4e59f09b16 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.150745 1718 raft_consensus.cc:491] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.150985 1718 raft_consensus.cc:513] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.152024 1718 leader_election.cc:290] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.152799 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c01b80fc8242d98fdfbf767212144d" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.153075 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c01b80fc8242d98fdfbf767212144d" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:31.153357 916 raft_consensus.cc:2463] T f8c01b80fc8242d98fdfbf767212144d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:31.154001 713 leader_election.cc:304] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
W20250112 02:52:31.154578 714 leader_election.cc:343] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.155009 1718 raft_consensus.cc:2798] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.155117 1718 raft_consensus.cc:491] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.155226 1718 raft_consensus.cc:3054] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.158666 1718 raft_consensus.cc:513] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.159914 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c01b80fc8242d98fdfbf767212144d" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:31.160724 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c01b80fc8242d98fdfbf767212144d" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:31.160988 916 raft_consensus.cc:3054] T f8c01b80fc8242d98fdfbf767212144d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.160719 1181 tablet_bootstrap.cc:492] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.163870 1181 ts_tablet_manager.cc:1397] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.057s user 0.025s sys 0.023s
W20250112 02:52:31.164182 714 leader_election.cc:343] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.164891 1181 raft_consensus.cc:357] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.165182 1181 raft_consensus.cc:383] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.165274 1181 raft_consensus.cc:738] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.165500 916 raft_consensus.cc:2463] T f8c01b80fc8242d98fdfbf767212144d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:31.166059 713 leader_election.cc:304] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.166436 1611 raft_consensus.cc:2798] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.166622 1611 raft_consensus.cc:695] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:31.166958 1611 consensus_queue.cc:237] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.169471 1634 raft_consensus.cc:491] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.169593 1181 consensus_queue.cc:260] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.169656 1634 raft_consensus.cc:513] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.170235 1181 ts_tablet_manager.cc:1428] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250112 02:52:31.170418 1634 leader_election.cc:290] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.170730 1181 tablet_bootstrap.cc:492] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.171298 1715 raft_consensus.cc:491] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.171447 1715 raft_consensus.cc:513] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
W20250112 02:52:31.171753 989 leader_election.cc:343] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: fb0aba0139cf479b80b32f3dfc936535
I20250112 02:52:31.172088 1715 leader_election.cc:290] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:31.172194 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0aba0139cf479b80b32f3dfc936535" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.172380 780 raft_consensus.cc:2463] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:31.172487 1181 tablet_bootstrap.cc:654] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.172623 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb6d20cfa0453e90e048825114525e" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.173501 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb6d20cfa0453e90e048825114525e" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.174228 1177 tablet_bootstrap.cc:492] T c04b64bc2f8348a086ed3c4e59f09b16 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:31.174417 991 leader_election.cc:304] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:31.174352 1177 ts_tablet_manager.cc:1397] T c04b64bc2f8348a086ed3c4e59f09b16 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.025s
I20250112 02:52:31.176777 1718 leader_election.cc:290] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.177273 1634 raft_consensus.cc:2798] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.177398 1634 raft_consensus.cc:491] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.177482 1634 raft_consensus.cc:3054] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.178889 1177 raft_consensus.cc:357] T c04b64bc2f8348a086ed3c4e59f09b16 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.179070 1177 raft_consensus.cc:383] T c04b64bc2f8348a086ed3c4e59f09b16 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.179157 1177 raft_consensus.cc:738] T c04b64bc2f8348a086ed3c4e59f09b16 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.179325 1177 consensus_queue.cc:260] T c04b64bc2f8348a086ed3c4e59f09b16 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.179812 1177 ts_tablet_manager.cc:1428] T c04b64bc2f8348a086ed3c4e59f09b16 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250112 02:52:31.180106 1177 tablet_bootstrap.cc:492] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:31.181161 1173 tablet_bootstrap.cc:492] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.181318 1173 ts_tablet_manager.cc:1397] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.058s user 0.010s sys 0.018s
I20250112 02:52:31.181676 1715 raft_consensus.cc:491] T 9f0314aa03b1473ebbe05ce73adf7838 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.181716 1177 tablet_bootstrap.cc:654] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.181807 1715 raft_consensus.cc:513] T 9f0314aa03b1473ebbe05ce73adf7838 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.181653 1634 raft_consensus.cc:513] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.182360 1715 leader_election.cc:290] T 9f0314aa03b1473ebbe05ce73adf7838 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
W20250112 02:52:31.183125 855 leader_election.cc:343] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.183243 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0aba0139cf479b80b32f3dfc936535" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:31.183411 779 raft_consensus.cc:3054] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.183740 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0314aa03b1473ebbe05ce73adf7838" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:31.184166 855 leader_election.cc:343] T 9f0314aa03b1473ebbe05ce73adf7838 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.185155 1173 raft_consensus.cc:357] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.185344 1173 raft_consensus.cc:383] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.185406 1173 raft_consensus.cc:738] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.185602 1173 consensus_queue.cc:260] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.186111 779 raft_consensus.cc:2463] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
W20250112 02:52:31.187016 854 leader_election.cc:343] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.187232 854 leader_election.cc:304] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.190601 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0314aa03b1473ebbe05ce73adf7838" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.191366 1173 ts_tablet_manager.cc:1428] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250112 02:52:31.195188 1173 tablet_bootstrap.cc:492] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:31.196239 991 leader_election.cc:304] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:31.196519 1714 raft_consensus.cc:2743] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.196751 1727 raft_consensus.cc:2798] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
W20250112 02:52:31.196841 854 leader_election.cc:343] T 9f0314aa03b1473ebbe05ce73adf7838 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.196897 1727 raft_consensus.cc:695] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:31.196949 854 leader_election.cc:304] T 9f0314aa03b1473ebbe05ce73adf7838 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.197119 1727 consensus_queue.cc:237] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.197603 1714 raft_consensus.cc:2743] T 9f0314aa03b1473ebbe05ce73adf7838 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.198817 1715 raft_consensus.cc:491] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.198958 1715 raft_consensus.cc:513] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.199600 1715 leader_election.cc:290] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:31.200266 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd9985cd20148f195a857c478b0755f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:31.200943 854 leader_election.cc:343] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.201256 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd9985cd20148f195a857c478b0755f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.201503 779 raft_consensus.cc:2463] T 7cd9985cd20148f195a857c478b0755f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:31.201964 855 leader_election.cc:304] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.202234 1715 raft_consensus.cc:2798] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.202332 1715 raft_consensus.cc:491] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.202409 1715 raft_consensus.cc:3054] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.202842 1634 leader_election.cc:290] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.205078 1715 raft_consensus.cc:513] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:31.206784 989 leader_election.cc:343] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: fb0aba0139cf479b80b32f3dfc936535
I20250112 02:52:31.207371 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd9985cd20148f195a857c478b0755f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:31.207588 779 raft_consensus.cc:3054] T 7cd9985cd20148f195a857c478b0755f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.210279 779 raft_consensus.cc:2463] T 7cd9985cd20148f195a857c478b0755f P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:31.210825 855 leader_election.cc:304] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:31.211086 1714 raft_consensus.cc:2798] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.211212 1714 raft_consensus.cc:695] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:31.211534 1173 tablet_bootstrap.cc:654] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.211453 1714 consensus_queue.cc:237] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.214143 1177 tablet_bootstrap.cc:492] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:31.214288 1177 ts_tablet_manager.cc:1397] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.019s
I20250112 02:52:31.215054 1177 raft_consensus.cc:357] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.215246 1177 raft_consensus.cc:383] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.215319 1177 raft_consensus.cc:738] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.215564 1177 consensus_queue.cc:260] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.216578 1181 tablet_bootstrap.cc:492] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.216702 1181 ts_tablet_manager.cc:1397] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.046s user 0.010s sys 0.002s
I20250112 02:52:31.216784 1177 ts_tablet_manager.cc:1428] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:31.217381 1181 raft_consensus.cc:357] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.217581 1181 raft_consensus.cc:383] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.217725 1181 raft_consensus.cc:738] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.217780 1177 tablet_bootstrap.cc:492] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:31.217918 1727 raft_consensus.cc:491] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.217919 1181 consensus_queue.cc:260] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.218116 1727 raft_consensus.cc:513] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.218448 1181 ts_tablet_manager.cc:1428] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:31.218744 1181 tablet_bootstrap.cc:492] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.218852 1727 leader_election.cc:290] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.219255 1177 tablet_bootstrap.cc:654] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:31.220144 989 leader_election.cc:343] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: e9fc9a4f1e8944abb76dad6c2ad53d35
I20250112 02:52:31.220479 1181 tablet_bootstrap.cc:654] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.221149 1611 consensus_queue.cc:1035] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:31.230448 1173 tablet_bootstrap.cc:492] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
W20250112 02:52:31.232188 991 leader_election.cc:343] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: e9fc9a4f1e8944abb76dad6c2ad53d35
I20250112 02:52:31.232307 991 leader_election.cc:304] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:31.232785 1727 raft_consensus.cc:2743] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.230587 1173 ts_tablet_manager.cc:1397] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.008s
I20250112 02:52:31.234369 1173 raft_consensus.cc:357] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.234539 1173 raft_consensus.cc:383] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.234617 1173 raft_consensus.cc:738] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.234810 1173 consensus_queue.cc:260] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.235309 1173 ts_tablet_manager.cc:1428] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:31.235565 1173 tablet_bootstrap.cc:492] T cc62dcbf67e84e36ae1fbe127ccd4879 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:31.237294 1173 tablet_bootstrap.cc:654] T cc62dcbf67e84e36ae1fbe127ccd4879 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.239044 1611 consensus_queue.cc:1035] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:31.244927 1181 tablet_bootstrap.cc:492] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.245074 1181 ts_tablet_manager.cc:1397] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20250112 02:52:31.245745 1181 raft_consensus.cc:357] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.245931 1181 raft_consensus.cc:383] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.245996 1181 raft_consensus.cc:738] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.246212 1181 consensus_queue.cc:260] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.246759 1181 ts_tablet_manager.cc:1428] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:31.247041 1181 tablet_bootstrap.cc:492] T 7ea075826b16475ca474e284fc6bc545 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.248739 1181 tablet_bootstrap.cc:654] T 7ea075826b16475ca474e284fc6bc545 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.253226 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd9985cd20148f195a857c478b0755f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:31.253940 854 leader_election.cc:343] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.254166 1715 leader_election.cc:290] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:31.256309 1177 tablet_bootstrap.cc:492] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:31.256368 1611 raft_consensus.cc:491] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.256484 1177 ts_tablet_manager.cc:1397] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.005s
I20250112 02:52:31.256544 1611 raft_consensus.cc:513] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:31.257129 1611 leader_election.cc:290] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:31.257422 1177 raft_consensus.cc:357] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.257637 1177 raft_consensus.cc:383] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.257745 1177 raft_consensus.cc:738] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.257882 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f135b226774285a077e5806131d91b" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:31.258078 919 raft_consensus.cc:2463] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:31.257994 1177 consensus_queue.cc:260] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.258730 713 leader_election.cc:304] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:31.259003 1611 raft_consensus.cc:2798] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.259094 1611 raft_consensus.cc:491] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.259182 1611 raft_consensus.cc:3054] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.259599 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f135b226774285a077e5806131d91b" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.260437 1177 ts_tablet_manager.cc:1428] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
W20250112 02:52:31.260527 714 leader_election.cc:343] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.260900 1177 tablet_bootstrap.cc:492] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:31.262617 1611 raft_consensus.cc:513] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:31.264704 1181 tablet_bootstrap.cc:492] T 7ea075826b16475ca474e284fc6bc545 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.264823 1181 ts_tablet_manager.cc:1397] T 7ea075826b16475ca474e284fc6bc545 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.004s
I20250112 02:52:31.265272 1611 leader_election.cc:290] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:31.265421 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f135b226774285a077e5806131d91b" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:31.265542 1181 raft_consensus.cc:357] T 7ea075826b16475ca474e284fc6bc545 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.265748 1181 raft_consensus.cc:383] T 7ea075826b16475ca474e284fc6bc545 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.265841 1181 raft_consensus.cc:738] T 7ea075826b16475ca474e284fc6bc545 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.266062 1181 consensus_queue.cc:260] T 7ea075826b16475ca474e284fc6bc545 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
W20250112 02:52:31.266508 714 leader_election.cc:343] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.266579 1181 ts_tablet_manager.cc:1428] T 7ea075826b16475ca474e284fc6bc545 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:31.266728 1177 tablet_bootstrap.cc:654] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.269415 1173 tablet_bootstrap.cc:492] T cc62dcbf67e84e36ae1fbe127ccd4879 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.269563 1173 ts_tablet_manager.cc:1397] T cc62dcbf67e84e36ae1fbe127ccd4879 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.007s
I20250112 02:52:31.270290 1173 raft_consensus.cc:357] T cc62dcbf67e84e36ae1fbe127ccd4879 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.270486 1173 raft_consensus.cc:383] T cc62dcbf67e84e36ae1fbe127ccd4879 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.270567 1173 raft_consensus.cc:738] T cc62dcbf67e84e36ae1fbe127ccd4879 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.270810 1173 consensus_queue.cc:260] T cc62dcbf67e84e36ae1fbe127ccd4879 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.271294 1173 ts_tablet_manager.cc:1428] T cc62dcbf67e84e36ae1fbe127ccd4879 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:31.271692 1181 tablet_bootstrap.cc:492] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.273296 1181 tablet_bootstrap.cc:654] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.274677 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f135b226774285a077e5806131d91b" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:31.274921 916 raft_consensus.cc:3054] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.281080 916 raft_consensus.cc:2463] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:31.281479 713 leader_election.cc:304] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.281857 1611 raft_consensus.cc:2798] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.281994 1611 raft_consensus.cc:695] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:31.282227 1611 consensus_queue.cc:237] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:31.284672 1181 tablet_bootstrap.cc:492] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.284869 1181 ts_tablet_manager.cc:1397] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250112 02:52:31.285861 1181 raft_consensus.cc:357] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.286142 1181 raft_consensus.cc:383] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.286238 1181 raft_consensus.cc:738] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.286631 1181 consensus_queue.cc:260] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.287447 1181 ts_tablet_manager.cc:1428] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250112 02:52:31.287871 1181 tablet_bootstrap.cc:492] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.289968 1181 tablet_bootstrap.cc:654] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.291752 1177 tablet_bootstrap.cc:492] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:31.291908 1177 ts_tablet_manager.cc:1397] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.008s
I20250112 02:52:31.292184 1173 tablet_bootstrap.cc:492] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:31.292629 1177 raft_consensus.cc:357] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.292861 1177 raft_consensus.cc:383] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.292932 1177 raft_consensus.cc:738] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.293157 1177 consensus_queue.cc:260] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.294425 1177 ts_tablet_manager.cc:1428] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:31.295038 1718 raft_consensus.cc:491] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.295276 1177 tablet_bootstrap.cc:492] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:31.295188 1718 raft_consensus.cc:513] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.295820 1718 leader_election.cc:290] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.296850 1177 tablet_bootstrap.cc:654] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.297092 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3ee5d8a17874d7ba54de51568e7d2e6" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:31.297168 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3ee5d8a17874d7ba54de51568e7d2e6" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:31.297675 714 leader_election.cc:343] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.297930 916 raft_consensus.cc:2463] T e3ee5d8a17874d7ba54de51568e7d2e6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:31.298323 713 leader_election.cc:304] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.298707 1718 raft_consensus.cc:2798] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.298789 1718 raft_consensus.cc:491] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.298880 1718 raft_consensus.cc:3054] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.301586 1718 raft_consensus.cc:513] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.302507 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3ee5d8a17874d7ba54de51568e7d2e6" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:31.302943 714 leader_election.cc:343] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.303385 1718 leader_election.cc:290] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.303398 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3ee5d8a17874d7ba54de51568e7d2e6" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:31.303606 919 raft_consensus.cc:3054] T e3ee5d8a17874d7ba54de51568e7d2e6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.303845 1181 tablet_bootstrap.cc:492] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.303966 1181 ts_tablet_manager.cc:1397] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.006s
I20250112 02:52:31.304678 1181 raft_consensus.cc:357] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.304891 1181 raft_consensus.cc:383] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.304982 1181 raft_consensus.cc:738] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.305181 1181 consensus_queue.cc:260] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.305642 1181 ts_tablet_manager.cc:1428] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:31.305891 1181 tablet_bootstrap.cc:492] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.306216 919 raft_consensus.cc:2463] T e3ee5d8a17874d7ba54de51568e7d2e6 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:31.307065 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:38771: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection refused (error 111),
I20250112 02:52:31.307206 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:52:31.307461 1181 tablet_bootstrap.cc:654] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.309331 1727 raft_consensus.cc:491] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.309463 1727 raft_consensus.cc:513] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.310065 1727 leader_election.cc:290] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:31.310976 991 leader_election.cc:343] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: 1726d61cc36a4deb83c5d604e8daf4f0
I20250112 02:52:31.313584 713 leader_election.cc:304] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.314934 1727 raft_consensus.cc:491] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.315078 1727 raft_consensus.cc:513] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.315651 1727 leader_election.cc:290] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.315938 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4916ae3917084cfe8c93240a8857be0e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.316113 779 raft_consensus.cc:2463] T 4916ae3917084cfe8c93240a8857be0e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:31.316110 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4916ae3917084cfe8c93240a8857be0e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:31.316287 916 raft_consensus.cc:2463] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:31.316496 991 leader_election.cc:304] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:31.316872 1727 raft_consensus.cc:2798] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.316956 1727 raft_consensus.cc:491] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.317041 1727 raft_consensus.cc:3054] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.318380 1611 raft_consensus.cc:2798] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.318511 1611 raft_consensus.cc:695] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
W20250112 02:52:31.319429 989 leader_election.cc:343] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 1726d61cc36a4deb83c5d604e8daf4f0
I20250112 02:52:31.319581 989 leader_election.cc:304] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:31.319573 1611 consensus_queue.cc:237] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.319837 1634 raft_consensus.cc:2743] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.320397 1727 raft_consensus.cc:513] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.321250 1173 tablet_bootstrap.cc:654] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.321290 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4916ae3917084cfe8c93240a8857be0e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:31.321457 779 raft_consensus.cc:3054] T 4916ae3917084cfe8c93240a8857be0e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.323789 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu
/tmp/dist-test-tasko6SHYo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:38771
--webserver_interface=127.0.104.61
--webserver_port=33823
--builtin_ntp_servers=127.0.104.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42341,127.0.104.61:38771,127.0.104.60:39987
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 02:52:31.324131 779 raft_consensus.cc:2463] T 4916ae3917084cfe8c93240a8857be0e P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:31.324575 991 leader_election.cc:304] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:31.324918 1634 raft_consensus.cc:2798] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.325084 1634 raft_consensus.cc:695] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:31.325376 1634 consensus_queue.cc:237] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.326268 1173 tablet_bootstrap.cc:492] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.326442 1173 ts_tablet_manager.cc:1397] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.005s
I20250112 02:52:31.327491 1727 leader_election.cc:290] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.328015 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4916ae3917084cfe8c93240a8857be0e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:31.328276 919 raft_consensus.cc:3054] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.328593 1718 raft_consensus.cc:491] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.328753 1718 raft_consensus.cc:513] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.329412 1718 leader_election.cc:290] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.331848 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfaad67d5e34c0a8338dccc0ef05466" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.332231 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfaad67d5e34c0a8338dccc0ef05466" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
W20250112 02:52:31.332309 714 leader_election.cc:343] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.332410 916 raft_consensus.cc:2463] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:31.332866 713 leader_election.cc:304] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.333355 1718 raft_consensus.cc:2798] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.333473 1718 raft_consensus.cc:491] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.333685 1718 raft_consensus.cc:3054] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.333912 919 raft_consensus.cc:2463] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:31.334693 1173 raft_consensus.cc:357] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:31.334865 1173 raft_consensus.cc:383] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.334946 1173 raft_consensus.cc:738] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.335161 1173 consensus_queue.cc:260] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:31.336336 1173 ts_tablet_manager.cc:1428] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250112 02:52:31.337421 1181 tablet_bootstrap.cc:492] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.337556 1181 ts_tablet_manager.cc:1397] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.003s
I20250112 02:52:31.338163 1181 raft_consensus.cc:357] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.338335 1181 raft_consensus.cc:383] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.338385 1181 raft_consensus.cc:738] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.338572 1181 consensus_queue.cc:260] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.338781 1173 tablet_bootstrap.cc:492] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:31.339082 1181 ts_tablet_manager.cc:1428] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 02:52:31.339378 1181 tablet_bootstrap.cc:492] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.341008 1181 tablet_bootstrap.cc:654] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.348556 1177 tablet_bootstrap.cc:492] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:31.348681 1181 tablet_bootstrap.cc:492] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.348768 1177 ts_tablet_manager.cc:1397] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.006s
I20250112 02:52:31.348795 1181 ts_tablet_manager.cc:1397] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20250112 02:52:31.349537 1181 raft_consensus.cc:357] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.349730 1181 raft_consensus.cc:383] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.349825 1181 raft_consensus.cc:738] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.350140 1173 tablet_bootstrap.cc:654] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.350052 1181 consensus_queue.cc:260] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.350647 1181 ts_tablet_manager.cc:1428] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:31.349905 1177 raft_consensus.cc:357] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.351042 1177 raft_consensus.cc:383] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.351131 1177 raft_consensus.cc:738] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.351361 1177 consensus_queue.cc:260] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.351713 1181 tablet_bootstrap.cc:492] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.353179 1181 tablet_bootstrap.cc:654] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.353950 1177 ts_tablet_manager.cc:1428] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 02:52:31.354213 1177 tablet_bootstrap.cc:492] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:31.357436 1718 raft_consensus.cc:513] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.359082 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfaad67d5e34c0a8338dccc0ef05466" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:31.359668 714 leader_election.cc:343] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.361842 1177 tablet_bootstrap.cc:654] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.362394 1718 leader_election.cc:290] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.362793 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfaad67d5e34c0a8338dccc0ef05466" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:31.363027 919 raft_consensus.cc:3054] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.363497 1181 tablet_bootstrap.cc:492] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.363637 1181 ts_tablet_manager.cc:1397] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.003s
I20250112 02:52:31.364300 1181 raft_consensus.cc:357] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:31.364522 1181 raft_consensus.cc:383] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.364589 1181 raft_consensus.cc:738] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.364794 1181 consensus_queue.cc:260] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:31.365275 1181 ts_tablet_manager.cc:1428] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:31.365545 1181 tablet_bootstrap.cc:492] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.366353 1634 raft_consensus.cc:491] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.366523 1634 raft_consensus.cc:513] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.367130 1634 leader_election.cc:290] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.367408 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd498ccce484a89855c982762086890" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:31.367945 989 leader_election.cc:343] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 3fd498ccce484a89855c982762086890
I20250112 02:52:31.368147 991 leader_election.cc:304] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:31.368557 1634 raft_consensus.cc:2743] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.368598 919 raft_consensus.cc:2463] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:31.368741 1634 raft_consensus.cc:491] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.368847 1634 raft_consensus.cc:513] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.368952 713 leader_election.cc:304] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.369279 1718 raft_consensus.cc:2798] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.369395 1634 leader_election.cc:290] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.369503 1718 raft_consensus.cc:695] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:31.369738 1718 consensus_queue.cc:237] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.370509 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f59617b34046568dd83bbb7cdc42d2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.370692 779 raft_consensus.cc:2463] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:31.371105 991 leader_election.cc:304] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:31.371384 1727 raft_consensus.cc:2798] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.371500 1727 raft_consensus.cc:491] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.371577 1727 raft_consensus.cc:3054] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.372022 1173 tablet_bootstrap.cc:492] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.372179 1173 ts_tablet_manager.cc:1397] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.002s
I20250112 02:52:31.373102 1173 raft_consensus.cc:357] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.373355 1173 raft_consensus.cc:383] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.373442 1173 raft_consensus.cc:738] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.373729 1173 consensus_queue.cc:260] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.374142 1727 raft_consensus.cc:513] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.374723 1727 leader_election.cc:290] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.375300 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f59617b34046568dd83bbb7cdc42d2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:31.375500 779 raft_consensus.cc:3054] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.378608 1173 ts_tablet_manager.cc:1428] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
W20250112 02:52:31.379212 989 leader_election.cc:343] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 35f59617b34046568dd83bbb7cdc42d2
I20250112 02:52:31.380097 1173 tablet_bootstrap.cc:492] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:31.381055 779 raft_consensus.cc:2463] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:31.381724 991 leader_election.cc:304] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:31.382292 1715 raft_consensus.cc:491] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.382431 1715 raft_consensus.cc:513] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.382666 1727 raft_consensus.cc:2798] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.382786 1727 raft_consensus.cc:695] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:31.383066 1715 leader_election.cc:290] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:31.383023 1727 consensus_queue.cc:237] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.383699 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea33e5c1a1fb46599c5836a36b130051" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.384392 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea33e5c1a1fb46599c5836a36b130051" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:31.384644 989 leader_election.cc:343] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 35f59617b34046568dd83bbb7cdc42d2
W20250112 02:52:31.384758 855 leader_election.cc:343] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:31.385057 854 leader_election.cc:343] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.385157 854 leader_election.cc:304] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.385450 1715 raft_consensus.cc:2743] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.389503 1181 tablet_bootstrap.cc:654] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.398279 1173 tablet_bootstrap.cc:654] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.410853 1727 raft_consensus.cc:491] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.411033 1727 raft_consensus.cc:513] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.411661 1727 leader_election.cc:290] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.412961 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadb2ab1d2024f889841ac91e065e064" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.413192 779 raft_consensus.cc:2463] T aadb2ab1d2024f889841ac91e065e064 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:31.413683 991 leader_election.cc:304] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
W20250112 02:52:31.414209 989 leader_election.cc:343] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: aadb2ab1d2024f889841ac91e065e064
I20250112 02:52:31.414533 1727 raft_consensus.cc:2798] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.414608 1727 raft_consensus.cc:491] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.414707 1727 raft_consensus.cc:3054] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.417452 1727 raft_consensus.cc:513] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:31.418179 713 consensus_peers.cc:487] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7bd96fb381f4e73a0081be3fc799c36. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:31.418390 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadb2ab1d2024f889841ac91e065e064" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:31.418581 779 raft_consensus.cc:3054] T aadb2ab1d2024f889841ac91e065e064 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.422200 779 raft_consensus.cc:2463] T aadb2ab1d2024f889841ac91e065e064 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:31.422608 991 leader_election.cc:304] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:31.422899 1634 raft_consensus.cc:2798] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.423063 1634 raft_consensus.cc:695] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:31.423316 1634 consensus_queue.cc:237] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.424088 1727 leader_election.cc:290] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:31.424616 989 leader_election.cc:343] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: aadb2ab1d2024f889841ac91e065e064
I20250112 02:52:31.425411 1715 raft_consensus.cc:491] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.425612 1715 raft_consensus.cc:513] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.426056 1394 raft_consensus.cc:1270] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 4 FOLLOWER]: Refusing update from remote peer 436220c23fa948efb5b6ca15875d8b0e: Log matching property violated. Preceding OpId in replica: term: 3 index: 501. Preceding OpId from leader: term: 4 index: 515. (index mismatch)
I20250112 02:52:31.426374 1715 leader_election.cc:290] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:31.426764 1787 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 515, Last known committed idx: 498, Time since last communication: 0.000s
I20250112 02:52:31.423310 1173 tablet_bootstrap.cc:492] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.427111 1173 ts_tablet_manager.cc:1397] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.047s user 0.001s sys 0.006s
I20250112 02:52:31.427546 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4e4f1e451b48dc9cdd780300f6888c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.427737 779 raft_consensus.cc:2463] T 7f4e4f1e451b48dc9cdd780300f6888c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:31.428107 855 leader_election.cc:304] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:31.428694 1181 tablet_bootstrap.cc:492] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.428822 1715 raft_consensus.cc:2798] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.428865 1181 ts_tablet_manager.cc:1397] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.006s
I20250112 02:52:31.428948 1715 raft_consensus.cc:491] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.429054 1715 raft_consensus.cc:3054] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.429605 1181 raft_consensus.cc:357] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.429797 1181 raft_consensus.cc:383] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.429898 1181 raft_consensus.cc:738] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.430119 1181 consensus_queue.cc:260] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.430928 1181 ts_tablet_manager.cc:1428] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:31.432807 1715 raft_consensus.cc:513] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.433761 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4e4f1e451b48dc9cdd780300f6888c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:31.433912 779 raft_consensus.cc:3054] T 7f4e4f1e451b48dc9cdd780300f6888c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.434032 1181 tablet_bootstrap.cc:492] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.436115 1181 tablet_bootstrap.cc:654] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.437281 779 raft_consensus.cc:2463] T 7f4e4f1e451b48dc9cdd780300f6888c P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:31.438280 855 leader_election.cc:304] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:31.438289 1173 raft_consensus.cc:357] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.438464 1173 raft_consensus.cc:383] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.438581 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4e4f1e451b48dc9cdd780300f6888c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.439050 1173 raft_consensus.cc:738] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.439253 1173 consensus_queue.cc:260] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.439859 1173 ts_tablet_manager.cc:1428] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20250112 02:52:31.440115 1173 tablet_bootstrap.cc:492] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:31.440383 1714 raft_consensus.cc:2798] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.440562 1714 raft_consensus.cc:695] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:31.440810 1714 consensus_queue.cc:237] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.441186 1177 tablet_bootstrap.cc:492] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:31.441334 1177 ts_tablet_manager.cc:1397] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.087s user 0.010s sys 0.024s
I20250112 02:52:31.441635 1173 tablet_bootstrap.cc:654] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.442044 1177 raft_consensus.cc:357] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.442225 1177 raft_consensus.cc:383] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.442291 1177 raft_consensus.cc:738] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.443208 1177 consensus_queue.cc:260] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
W20250112 02:52:31.445118 854 leader_election.cc:343] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.445935 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4e4f1e451b48dc9cdd780300f6888c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:31.446555 854 leader_election.cc:343] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.446764 1715 leader_election.cc:290] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:31.448407 1177 ts_tablet_manager.cc:1428] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250112 02:52:31.451984 1177 tablet_bootstrap.cc:492] T 8a12b225131b464494a9f02133d37aea P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:31.454032 1177 tablet_bootstrap.cc:654] T 8a12b225131b464494a9f02133d37aea P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.455940 1714 raft_consensus.cc:491] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.456113 1714 raft_consensus.cc:513] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:31.456848 1714 leader_election.cc:290] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:31.457000 1714 raft_consensus.cc:491] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.457144 1714 raft_consensus.cc:513] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.457850 1714 leader_election.cc:290] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:31.457929 1173 tablet_bootstrap.cc:492] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.458066 1173 ts_tablet_manager.cc:1397] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.002s
I20250112 02:52:31.458557 1050 raft_consensus.cc:3054] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.458798 1173 raft_consensus.cc:357] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.459000 1173 raft_consensus.cc:383] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.459064 1173 raft_consensus.cc:738] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.459630 1173 consensus_queue.cc:260] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.460536 1173 ts_tablet_manager.cc:1428] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:31.460821 1173 tablet_bootstrap.cc:492] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:31.463104 1611 consensus_queue.cc:1035] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:31.465344 1173 tablet_bootstrap.cc:654] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.472723 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6913b0ac9ceb4dbc9cdf591f159134f7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.473023 780 raft_consensus.cc:2463] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:31.473862 855 leader_election.cc:304] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:31.474059 1611 consensus_queue.cc:1035] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:31.475107 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ccb88108f24e36ba4b7ebf36398950" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.475679 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6913b0ac9ceb4dbc9cdf591f159134f7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:31.475867 854 leader_election.cc:343] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:31.476330 854 leader_election.cc:343] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.477252 1714 raft_consensus.cc:2798] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.477393 1714 raft_consensus.cc:491] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.477551 1714 raft_consensus.cc:3054] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.485790 1609 consensus_queue.cc:1035] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:31.490171 1181 tablet_bootstrap.cc:492] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.490341 1181 ts_tablet_manager.cc:1397] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.056s user 0.010s sys 0.006s
I20250112 02:52:31.491132 1181 raft_consensus.cc:357] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.491390 1181 raft_consensus.cc:383] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.491485 1181 raft_consensus.cc:738] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.491706 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ccb88108f24e36ba4b7ebf36398950" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.491729 1181 consensus_queue.cc:260] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.492297 1181 ts_tablet_manager.cc:1428] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 02:52:31.492619 1181 tablet_bootstrap.cc:492] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.491397 1791 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 436220c23fa948efb5b6ca15875d8b0e. Latest consensus state: current_term: 4 leader_uuid: "436220c23fa948efb5b6ca15875d8b0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:31.493386 1791 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:52:31.498626 1181 tablet_bootstrap.cc:654] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.499037 1791 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "436220c23fa948efb5b6ca15875d8b0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:31.499198 1791 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [sys.catalog]: This master's current role is: FOLLOWER
W20250112 02:52:31.499361 439 consensus_peers.cc:487] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e -> Peer f4a49a5756b143fcb767c3f41463f8d9 (127.0.104.61:38771): Couldn't send request to peer f4a49a5756b143fcb767c3f41463f8d9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:31.502436 1173 tablet_bootstrap.cc:492] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.502563 1173 ts_tablet_manager.cc:1397] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.042s user 0.001s sys 0.005s
I20250112 02:52:31.503232 1173 raft_consensus.cc:357] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:31.503400 1173 raft_consensus.cc:383] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.503504 1173 raft_consensus.cc:738] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.503715 1173 consensus_queue.cc:260] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:31.504189 1173 ts_tablet_manager.cc:1428] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:31.504503 1173 tablet_bootstrap.cc:492] T 1c639a2bcd4740e4be7b2b162004669a P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:31.504880 855 leader_election.cc:343] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.505378 855 leader_election.cc:304] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.506289 1173 tablet_bootstrap.cc:654] T 1c639a2bcd4740e4be7b2b162004669a P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.508335 1714 raft_consensus.cc:513] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.508935 1634 raft_consensus.cc:491] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.509101 1634 raft_consensus.cc:513] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.509732 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6913b0ac9ceb4dbc9cdf591f159134f7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:31.510075 1634 leader_election.cc:290] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.510126 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7bd96fb381f4e73a0081be3fc799c36" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:31.510890 854 leader_election.cc:343] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:31.511435 989 leader_election.cc:343] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: b7bd96fb381f4e73a0081be3fc799c36
I20250112 02:52:31.511595 989 leader_election.cc:304] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:31.511950 1714 leader_election.cc:290] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:31.512111 1714 raft_consensus.cc:2743] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.512352 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6913b0ac9ceb4dbc9cdf591f159134f7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:31.512567 779 raft_consensus.cc:3054] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.513152 1634 raft_consensus.cc:2743] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.514343 1177 tablet_bootstrap.cc:492] T 8a12b225131b464494a9f02133d37aea P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:31.514546 1177 ts_tablet_manager.cc:1397] T 8a12b225131b464494a9f02133d37aea P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.066s user 0.011s sys 0.013s
I20250112 02:52:31.515450 1177 raft_consensus.cc:357] T 8a12b225131b464494a9f02133d37aea P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.515748 1177 raft_consensus.cc:383] T 8a12b225131b464494a9f02133d37aea P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.515882 1177 raft_consensus.cc:738] T 8a12b225131b464494a9f02133d37aea P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.516167 1177 consensus_queue.cc:260] T 8a12b225131b464494a9f02133d37aea P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.516822 1177 ts_tablet_manager.cc:1428] T 8a12b225131b464494a9f02133d37aea P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 02:52:31.519347 1177 tablet_bootstrap.cc:492] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:31.521311 1177 tablet_bootstrap.cc:654] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.521740 1634 raft_consensus.cc:491] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.521941 1634 raft_consensus.cc:513] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:31.522984 1634 leader_election.cc:290] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.523324 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f135b226774285a077e5806131d91b" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:31.523588 916 raft_consensus.cc:2388] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1e775296eb746db8b318f17d26820e6 in current term 1: Already voted for candidate 289afc116aa74408aa5bff74e11c50d8 in this term.
I20250112 02:52:31.523670 1051 raft_consensus.cc:3054] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.525259 1715 raft_consensus.cc:491] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.525408 1173 tablet_bootstrap.cc:492] T 1c639a2bcd4740e4be7b2b162004669a P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.525439 1715 raft_consensus.cc:513] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.525650 1173 ts_tablet_manager.cc:1397] T 1c639a2bcd4740e4be7b2b162004669a P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250112 02:52:31.526593 1173 raft_consensus.cc:357] T 1c639a2bcd4740e4be7b2b162004669a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:31.526880 1173 raft_consensus.cc:383] T 1c639a2bcd4740e4be7b2b162004669a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.526974 1173 raft_consensus.cc:738] T 1c639a2bcd4740e4be7b2b162004669a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.527009 1715 leader_election.cc:290] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:31.527248 1173 consensus_queue.cc:260] T 1c639a2bcd4740e4be7b2b162004669a P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:31.527858 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85300a9aa1c4c50a3957a004bea7933" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.528115 1173 ts_tablet_manager.cc:1428] T 1c639a2bcd4740e4be7b2b162004669a P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250112 02:52:31.528442 1173 tablet_bootstrap.cc:492] T 7cd9985cd20148f195a857c478b0755f P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:31.528492 854 leader_election.cc:343] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.529876 1173 tablet_bootstrap.cc:654] T 7cd9985cd20148f195a857c478b0755f P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.522850 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f135b226774285a077e5806131d91b" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.526731 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85300a9aa1c4c50a3957a004bea7933" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.526927 1602 consensus_queue.cc:1035] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:31.537818 991 leader_election.cc:304] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:31.527400 779 raft_consensus.cc:2463] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
W20250112 02:52:31.537884 855 leader_election.cc:343] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.538008 855 leader_election.cc:304] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.538285 1634 raft_consensus.cc:3054] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.538417 855 leader_election.cc:304] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.538754 1715 raft_consensus.cc:2743] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.539386 1715 raft_consensus.cc:2798] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.539817 1715 raft_consensus.cc:695] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:31.540122 1715 consensus_queue.cc:237] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.540952 1634 raft_consensus.cc:2743] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.544095 1181 tablet_bootstrap.cc:492] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.544250 1181 ts_tablet_manager.cc:1397] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.014s
I20250112 02:52:31.544921 1181 raft_consensus.cc:357] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.545096 1181 raft_consensus.cc:383] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.545177 1181 raft_consensus.cc:738] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.545356 1181 consensus_queue.cc:260] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.545993 1181 ts_tablet_manager.cc:1428] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:31.546236 1173 tablet_bootstrap.cc:492] T 7cd9985cd20148f195a857c478b0755f P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.546290 1181 tablet_bootstrap.cc:492] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.546351 1173 ts_tablet_manager.cc:1397] T 7cd9985cd20148f195a857c478b0755f P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.001s
I20250112 02:52:31.547122 1173 raft_consensus.cc:357] T 7cd9985cd20148f195a857c478b0755f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.547330 1173 raft_consensus.cc:383] T 7cd9985cd20148f195a857c478b0755f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.547425 1173 raft_consensus.cc:738] T 7cd9985cd20148f195a857c478b0755f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.547983 1181 tablet_bootstrap.cc:654] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:31.550236 713 consensus_peers.cc:487] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fd498ccce484a89855c982762086890. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:31.555675 1173 consensus_queue.cc:260] T 7cd9985cd20148f195a857c478b0755f P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.556609 1173 ts_tablet_manager.cc:1428] T 7cd9985cd20148f195a857c478b0755f P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250112 02:52:31.556955 1173 tablet_bootstrap.cc:492] T 161fe26b9c0e4543b7aeb0021506d72a P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:31.558871 1173 tablet_bootstrap.cc:654] T 161fe26b9c0e4543b7aeb0021506d72a P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.560427 1714 raft_consensus.cc:491] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.560593 1177 tablet_bootstrap.cc:492] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:31.560619 1714 raft_consensus.cc:513] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.560807 1177 ts_tablet_manager.cc:1397] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.007s
I20250112 02:52:31.561604 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8a178bafcb457f9b0e9587df560493" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.561681 1177 raft_consensus.cc:357] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.561892 1177 raft_consensus.cc:383] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.561993 1177 raft_consensus.cc:738] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
W20250112 02:52:31.562093 854 leader_election.cc:343] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.562376 1714 leader_election.cc:290] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:31.562291 1177 consensus_queue.cc:260] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.562721 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8a178bafcb457f9b0e9587df560493" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:31.563189 855 leader_election.cc:343] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.563295 855 leader_election.cc:304] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.563791 1714 raft_consensus.cc:2743] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.565058 1177 ts_tablet_manager.cc:1428] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20250112 02:52:31.565438 1177 tablet_bootstrap.cc:492] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:31.566293 1602 raft_consensus.cc:491] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.566416 1602 raft_consensus.cc:513] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.567009 1602 leader_election.cc:290] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.567919 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc62dcbf67e84e36ae1fbe127ccd4879" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.568110 1054 raft_consensus.cc:2463] T cc62dcbf67e84e36ae1fbe127ccd4879 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:31.570365 714 leader_election.cc:304] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:31.570672 1602 raft_consensus.cc:2798] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.570788 1602 raft_consensus.cc:491] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.570855 1602 raft_consensus.cc:3054] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.573333 1602 raft_consensus.cc:513] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.574193 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc62dcbf67e84e36ae1fbe127ccd4879" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:31.574283 1602 leader_election.cc:290] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.574409 1054 raft_consensus.cc:3054] T cc62dcbf67e84e36ae1fbe127ccd4879 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.576588 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc62dcbf67e84e36ae1fbe127ccd4879" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:31.576584 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc62dcbf67e84e36ae1fbe127ccd4879" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:31.576795 916 raft_consensus.cc:3054] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.577652 1054 raft_consensus.cc:2463] T cc62dcbf67e84e36ae1fbe127ccd4879 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:31.578069 714 leader_election.cc:304] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:31.578423 1602 raft_consensus.cc:2798] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.578804 1602 raft_consensus.cc:695] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:31.579200 1602 consensus_queue.cc:237] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.578717 1181 tablet_bootstrap.cc:492] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.579653 1181 ts_tablet_manager.cc:1397] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.021s
I20250112 02:52:31.579267 916 raft_consensus.cc:2463] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:31.580644 1181 raft_consensus.cc:357] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.580915 1181 raft_consensus.cc:383] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.581009 1181 raft_consensus.cc:738] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.581290 1181 consensus_queue.cc:260] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.582000 1181 ts_tablet_manager.cc:1428] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:31.582331 1181 tablet_bootstrap.cc:492] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.584450 1181 tablet_bootstrap.cc:654] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.589847 1609 consensus_queue.cc:1035] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:31.591385 1782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "436220c23fa948efb5b6ca15875d8b0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:31.591635 1782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [sys.catalog]: This master's current role is: LEADER
I20250112 02:52:31.597095 1715 raft_consensus.cc:491] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.597257 1715 raft_consensus.cc:513] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.598156 1715 leader_election.cc:290] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:31.598094 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daef3a58605e40978f7389698b22265d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.598397 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daef3a58605e40978f7389698b22265d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:31.598798 854 leader_election.cc:343] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:31.599018 855 leader_election.cc:343] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.599143 855 leader_election.cc:304] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.600051 1634 raft_consensus.cc:491] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.600260 1634 raft_consensus.cc:513] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.601006 1634 leader_election.cc:290] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.601594 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a88463991294bd8a335337f8b072c3d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.601945 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a88463991294bd8a335337f8b072c3d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:31.602391 989 leader_election.cc:304] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:31.602816 1634 raft_consensus.cc:2743] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.605721 1715 raft_consensus.cc:2743] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.609846 1734 raft_consensus.cc:491] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.610136 1734 raft_consensus.cc:513] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.611156 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039a4734358646508093200d59d30013" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.611681 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039a4734358646508093200d59d30013" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:31.611912 919 raft_consensus.cc:2463] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
W20250112 02:52:31.612320 714 leader_election.cc:343] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.612639 1181 tablet_bootstrap.cc:492] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.612651 713 leader_election.cc:304] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.612946 1734 leader_election.cc:290] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.612977 1718 raft_consensus.cc:2798] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.613085 1718 raft_consensus.cc:491] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.613199 1718 raft_consensus.cc:3054] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.615362 1706 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 02:52:31.616457 1718 raft_consensus.cc:513] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.617623 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039a4734358646508093200d59d30013" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:31.618207 714 leader_election.cc:343] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.619211 1718 leader_election.cc:290] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.620352 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039a4734358646508093200d59d30013" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:31.620601 916 raft_consensus.cc:3054] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.624061 916 raft_consensus.cc:2463] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:31.624719 713 leader_election.cc:304] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.625325 1611 raft_consensus.cc:2798] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.625510 1611 raft_consensus.cc:695] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:31.625799 1611 consensus_queue.cc:237] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.624279 1181 ts_tablet_manager.cc:1397] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.002s
I20250112 02:52:31.633236 1181 raft_consensus.cc:357] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.634312 1181 raft_consensus.cc:383] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.634644 1181 raft_consensus.cc:738] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.635083 1181 consensus_queue.cc:260] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.637301 1181 ts_tablet_manager.cc:1428] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250112 02:52:31.637840 1181 tablet_bootstrap.cc:492] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.640467 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]
I20250112 02:52:31.642668 1181 tablet_bootstrap.cc:654] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.647272 1173 tablet_bootstrap.cc:492] T 161fe26b9c0e4543b7aeb0021506d72a P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.647444 1173 ts_tablet_manager.cc:1397] T 161fe26b9c0e4543b7aeb0021506d72a P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.091s user 0.024s sys 0.052s
I20250112 02:52:31.648156 1173 raft_consensus.cc:357] T 161fe26b9c0e4543b7aeb0021506d72a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.648352 1173 raft_consensus.cc:383] T 161fe26b9c0e4543b7aeb0021506d72a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.648427 1173 raft_consensus.cc:738] T 161fe26b9c0e4543b7aeb0021506d72a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.648675 1173 consensus_queue.cc:260] T 161fe26b9c0e4543b7aeb0021506d72a P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.649331 1173 ts_tablet_manager.cc:1428] T 161fe26b9c0e4543b7aeb0021506d72a P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:31.649665 1173 tablet_bootstrap.cc:492] T e3ee5d8a17874d7ba54de51568e7d2e6 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:31.649659 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_7921b99e24f84cceb58128f18925ad49 [id=0525f87eccbb4a1d8726d18a00f9a4e4]
I20250112 02:52:31.651079 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_c0aaadb5e69e46e8983f4d289a44afe2 [id=08299ab225a94e37a67722107a5ff4ce]
I20250112 02:52:31.651320 1173 tablet_bootstrap.cc:654] T e3ee5d8a17874d7ba54de51568e7d2e6 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.651360 1634 raft_consensus.cc:491] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.651563 1634 raft_consensus.cc:513] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.652606 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e166b3f8414ababbd4a3f07fb90c4c" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:31.652613 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e166b3f8414ababbd4a3f07fb90c4c" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.652920 916 raft_consensus.cc:2388] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1e775296eb746db8b318f17d26820e6 in current term 1: Already voted for candidate 289afc116aa74408aa5bff74e11c50d8 in this term.
I20250112 02:52:31.653375 1634 leader_election.cc:290] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.653504 989 leader_election.cc:304] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:31.653932 1634 raft_consensus.cc:3054] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.655654 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_3d820aeff9534cc98f5722887aed5300 [id=08a760dcd20642fb869437d4ccc3f333]
I20250112 02:52:31.656271 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]
I20250112 02:52:31.656551 1634 raft_consensus.cc:2743] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.661329 1181 tablet_bootstrap.cc:492] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.661489 1181 ts_tablet_manager.cc:1397] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.002s
I20250112 02:52:31.661942 1054 raft_consensus.cc:3054] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.663779 1177 tablet_bootstrap.cc:654] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.664022 1173 tablet_bootstrap.cc:492] T e3ee5d8a17874d7ba54de51568e7d2e6 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.664127 1173 ts_tablet_manager.cc:1397] T e3ee5d8a17874d7ba54de51568e7d2e6 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.007s
I20250112 02:52:31.664772 1173 raft_consensus.cc:357] T e3ee5d8a17874d7ba54de51568e7d2e6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.664961 1173 raft_consensus.cc:383] T e3ee5d8a17874d7ba54de51568e7d2e6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.665048 1173 raft_consensus.cc:738] T e3ee5d8a17874d7ba54de51568e7d2e6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.665236 1173 consensus_queue.cc:260] T e3ee5d8a17874d7ba54de51568e7d2e6 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.665416 1602 consensus_queue.cc:1035] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:31.665781 1173 ts_tablet_manager.cc:1428] T e3ee5d8a17874d7ba54de51568e7d2e6 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:31.665794 1181 raft_consensus.cc:357] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.666055 1173 tablet_bootstrap.cc:492] T c04b64bc2f8348a086ed3c4e59f09b16 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:31.666908 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_9f27fd5786f84a8084cdf01f95f4519b [id=0e7297e807b545fa8482864ab59eeb41]
I20250112 02:52:31.667273 1181 raft_consensus.cc:383] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.667575 1181 raft_consensus.cc:738] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.668076 1173 tablet_bootstrap.cc:654] T c04b64bc2f8348a086ed3c4e59f09b16 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.668555 1181 consensus_queue.cc:260] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.670317 1181 ts_tablet_manager.cc:1428] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250112 02:52:31.671787 1181 tablet_bootstrap.cc:492] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.672174 1715 raft_consensus.cc:491] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.672341 1715 raft_consensus.cc:513] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.673044 1715 leader_election.cc:290] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:31.673463 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04b64bc2f8348a086ed3c4e59f09b16" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.673581 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_f0fb142b9aab490bb067d3cf64dc47b1 [id=1991c0821ef4478eb0eb9282ab772c0c]
I20250112 02:52:31.673580 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04b64bc2f8348a086ed3c4e59f09b16" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.673784 779 raft_consensus.cc:2463] T c04b64bc2f8348a086ed3c4e59f09b16 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:31.674222 855 leader_election.cc:304] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:31.674584 1715 raft_consensus.cc:2798] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.674680 1715 raft_consensus.cc:491] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.674722 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_692d9e1bfcfd435e8c420696c1e5aae5 [id=1ec97dc2b50a4a3dbbc449559938cead]
W20250112 02:52:31.674695 854 leader_election.cc:343] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.674788 1715 raft_consensus.cc:3054] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.675158 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_328664e3c01f483ca8e5ff50a497f68e [id=1f5c5f9f5e664250898a3935787b6b51]
I20250112 02:52:31.675616 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_6e77252e817f43399c9b3f5b221795ad [id=1fa1474ab8804667b22e5f30c48c1630]
I20250112 02:52:31.676072 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_e1d4a3104fc0481287acede4261941c6 [id=231e339ce95d48c99307e09b9de230c9]
I20250112 02:52:31.676687 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]
I20250112 02:52:31.677385 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_83177e7dc14645dbb0e5ef573e80e72f [id=2c077da077834ae4bcc84efdd9e89120]
I20250112 02:52:31.677868 1715 raft_consensus.cc:513] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.678712 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04b64bc2f8348a086ed3c4e59f09b16" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:31.678897 779 raft_consensus.cc:3054] T c04b64bc2f8348a086ed3c4e59f09b16 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.680410 1173 tablet_bootstrap.cc:492] T c04b64bc2f8348a086ed3c4e59f09b16 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.680537 1173 ts_tablet_manager.cc:1397] T c04b64bc2f8348a086ed3c4e59f09b16 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20250112 02:52:31.681182 1173 raft_consensus.cc:357] T c04b64bc2f8348a086ed3c4e59f09b16 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.681382 1173 raft_consensus.cc:383] T c04b64bc2f8348a086ed3c4e59f09b16 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.681459 1173 raft_consensus.cc:738] T c04b64bc2f8348a086ed3c4e59f09b16 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.681619 779 raft_consensus.cc:2463] T c04b64bc2f8348a086ed3c4e59f09b16 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:31.681675 1173 consensus_queue.cc:260] T c04b64bc2f8348a086ed3c4e59f09b16 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.682029 855 leader_election.cc:304] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:31.682224 1173 ts_tablet_manager.cc:1428] T c04b64bc2f8348a086ed3c4e59f09b16 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:31.682277 1714 raft_consensus.cc:2798] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.682390 1714 raft_consensus.cc:695] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:31.682487 1173 tablet_bootstrap.cc:492] T 015412a5fa304453bd8d1be1c3cd7a46 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:31.682621 1714 consensus_queue.cc:237] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.684237 1173 tablet_bootstrap.cc:654] T 015412a5fa304453bd8d1be1c3cd7a46 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.684445 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_045e5b104dc64217aa39c22534007cab [id=2c0e94e8c5774da1b8130c7651d94500]
I20250112 02:52:31.686182 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_829049fe22b74e62a87dd1bed65fb3fe [id=2e892eba68234ffca0be4ef3e49cf309]
I20250112 02:52:31.687510 1173 tablet_bootstrap.cc:492] T 015412a5fa304453bd8d1be1c3cd7a46 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.675434 1181 tablet_bootstrap.cc:654] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.687747 1173 ts_tablet_manager.cc:1397] T 015412a5fa304453bd8d1be1c3cd7a46 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20250112 02:52:31.688516 1715 leader_election.cc:290] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:31.688444 1173 raft_consensus.cc:357] T 015412a5fa304453bd8d1be1c3cd7a46 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.688611 1173 raft_consensus.cc:383] T 015412a5fa304453bd8d1be1c3cd7a46 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.688688 1173 raft_consensus.cc:738] T 015412a5fa304453bd8d1be1c3cd7a46 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.688900 1173 consensus_queue.cc:260] T 015412a5fa304453bd8d1be1c3cd7a46 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.689822 1173 ts_tablet_manager.cc:1428] T 015412a5fa304453bd8d1be1c3cd7a46 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 02:52:31.690055 1173 tablet_bootstrap.cc:492] T 7f4e4f1e451b48dc9cdd780300f6888c P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:31.692147 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04b64bc2f8348a086ed3c4e59f09b16" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:31.692350 1050 raft_consensus.cc:3054] T c04b64bc2f8348a086ed3c4e59f09b16 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.693097 1177 tablet_bootstrap.cc:492] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:31.693230 1177 ts_tablet_manager.cc:1397] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.128s user 0.008s sys 0.004s
I20250112 02:52:31.693987 1177 raft_consensus.cc:357] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.694175 1177 raft_consensus.cc:383] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.694243 1177 raft_consensus.cc:738] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.694424 1177 consensus_queue.cc:260] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.694691 1173 tablet_bootstrap.cc:654] T 7f4e4f1e451b48dc9cdd780300f6888c P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.694979 1177 ts_tablet_manager.cc:1428] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:31.698611 1177 tablet_bootstrap.cc:492] T ed7243c6329d4f11aa25eea2b7b67683 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:31.695854 1050 raft_consensus.cc:2463] T c04b64bc2f8348a086ed3c4e59f09b16 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:31.704581 1611 consensus_queue.cc:1035] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:31.704964 1611 consensus_queue.cc:1035] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:31.705294 1611 raft_consensus.cc:491] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.705399 1611 raft_consensus.cc:513] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.706298 1611 leader_election.cc:290] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.707347 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161fe26b9c0e4543b7aeb0021506d72a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:31.707589 919 raft_consensus.cc:2463] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:31.708727 713 leader_election.cc:304] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:31.709051 1611 raft_consensus.cc:2798] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.709162 1611 raft_consensus.cc:491] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.709235 1611 raft_consensus.cc:3054] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.713135 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161fe26b9c0e4543b7aeb0021506d72a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.713356 1050 raft_consensus.cc:2463] T 161fe26b9c0e4543b7aeb0021506d72a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:31.718755 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_0dec5de812ab4ea48f1b7a8f4cb0cc4f [id=2f43ebd749304c0287769224ac6b12c7]
I20250112 02:52:31.719684 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_f9db9e236376446f8210cc4c9708ec6a [id=2fa03135646d4939b6efc7811ee4df10]
I20250112 02:52:31.720293 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_e14b6c6d907942ebb7f8410e18d38ce2 [id=3097d5437937423e95b710c00473b6fa]
I20250112 02:52:31.722795 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_672d7452ad6d4c99ae9f2ae77b592f78 [id=33dad7ab4106404aaf33266ed1ea5635]
I20250112 02:52:31.723223 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6]
I20250112 02:52:31.724515 1181 tablet_bootstrap.cc:492] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.724695 1181 ts_tablet_manager.cc:1397] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.001s
I20250112 02:52:31.725930 1181 raft_consensus.cc:357] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.726357 1181 raft_consensus.cc:383] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.726526 1181 raft_consensus.cc:738] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.727149 1181 consensus_queue.cc:260] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.730190 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_e8e62bd2fffe4f50849f722dbece2a2c [id=3603f999c77a48dca737bc8ff74eb9de]
I20250112 02:52:31.731505 1181 ts_tablet_manager.cc:1428] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20250112 02:52:31.731948 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_e1b086de31974b33b7ac824ebeb757ee [id=36b5d7e7c4444808921063d46a50da46]
I20250112 02:52:31.733834 1181 tablet_bootstrap.cc:492] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.734151 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]
I20250112 02:52:31.734689 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_849b7f79a5784f5fabed0136deb0a129 [id=386e3565dbf2490997e5cc81d6f02567]
I20250112 02:52:31.735337 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_150748c55de444749d147c320c6d8c0c [id=39ac22eea1dd45de96d7e1829ad9844d]
I20250112 02:52:31.737141 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_ed3c26ab0711404581dcd62adb343dde [id=3a0f5ac3329d4b60befce41c91f5ae5c]
I20250112 02:52:31.737825 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_5adbd32e9a0c4736844af4c72b7787fd [id=3a629cb4a4104e30b0beb8f729ba0a22]
I20250112 02:52:31.738493 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_f7fd6e154c104f7ab66a01e7f8202da6 [id=4447b2d49c864e268d7eec91384899cc]
I20250112 02:52:31.739009 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_a56064e2c6b449b7b1e32f2a69f2f7fe [id=448877a19b0d4ed2abbc3a796e40eaeb]
I20250112 02:52:31.739595 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_79dbb2fcd6fe4c939bc0637b4ddf1b1e [id=44975f62b4f9429981075254868cb170]
I20250112 02:52:31.740083 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_56ea07abda424090a7942efabc871092 [id=44f7673679d14145a8cf65abd82094ce]
I20250112 02:52:31.740883 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_9487626cbda14256af1f17d2675b5d33 [id=485dd281891f45e8b407b306572d6e29]
I20250112 02:52:31.741904 1181 tablet_bootstrap.cc:654] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.742426 1173 tablet_bootstrap.cc:492] T 7f4e4f1e451b48dc9cdd780300f6888c P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.742574 1173 ts_tablet_manager.cc:1397] T 7f4e4f1e451b48dc9cdd780300f6888c P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.053s user 0.016s sys 0.024s
I20250112 02:52:31.743109 1714 raft_consensus.cc:491] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.743264 1714 raft_consensus.cc:513] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.743256 1173 raft_consensus.cc:357] T 7f4e4f1e451b48dc9cdd780300f6888c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.743448 1173 raft_consensus.cc:383] T 7f4e4f1e451b48dc9cdd780300f6888c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.743553 1173 raft_consensus.cc:738] T 7f4e4f1e451b48dc9cdd780300f6888c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.743781 1173 consensus_queue.cc:260] T 7f4e4f1e451b48dc9cdd780300f6888c P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.743916 1714 leader_election.cc:290] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
W20250112 02:52:31.744904 713 consensus_peers.cc:487] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9199b0d2b924a79938a979df4196bb4. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:31.745056 1054 raft_consensus.cc:3054] T 55fb1a197d7e40689ccad82a44492371 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.748049 1714 raft_consensus.cc:491] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.748173 1714 raft_consensus.cc:513] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.748732 1714 leader_election.cc:290] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:31.748826 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_b0d877b624604ddc93c56bec6e05261f [id=510ea5b8b1ae461faaa7c9787162f29a]
I20250112 02:52:31.749200 1634 raft_consensus.cc:491] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.749321 1634 raft_consensus.cc:513] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.749447 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_73e52096e12e47bab61cbfd2c72c70b1 [id=5424d9b98fae48949ed175c5498d24e1]
I20250112 02:52:31.749850 1634 leader_election.cc:290] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.750056 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]
I20250112 02:52:31.750699 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23e7a2e88c374e949ced92c557225cbf" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:31.750988 1173 ts_tablet_manager.cc:1428] T 7f4e4f1e451b48dc9cdd780300f6888c P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250112 02:52:31.751490 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61593969d2234ce3a6089027b067d922" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.752265 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d0f67141dc46b6a87ca0ad080d5bfe" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.752545 1718 consensus_queue.cc:1035] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20250112 02:52:31.752969 854 leader_election.cc:343] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.753050 1173 tablet_bootstrap.cc:492] T 039a4734358646508093200d59d30013 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:31.753479 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d0f67141dc46b6a87ca0ad080d5bfe" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:31.753983 854 leader_election.cc:343] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.754673 1173 tablet_bootstrap.cc:654] T 039a4734358646508093200d59d30013 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:31.755026 855 leader_election.cc:343] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.755142 855 leader_election.cc:304] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.755343 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_03ccb002507e4b93b1b413ae70602cfc [id=594044c9b27a441cb1002a7b35a6a846]
I20250112 02:52:31.755797 1714 raft_consensus.cc:2743] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.756070 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_4934f6022f2441eda88751236a5798be [id=5c4769bfcefd4fe0a0f3c2e315d50c32]
I20250112 02:52:31.756585 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_24cecfd421cd4047972cc0a6255ef0c0 [id=5cec72cf833347f3bfbca061f8feca41]
I20250112 02:52:31.758631 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_03e3448d809c4fc792d0bff94a713044 [id=5f3cc73b47414c438591107e35a12472]
I20250112 02:52:31.761317 1181 tablet_bootstrap.cc:492] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.761457 1181 ts_tablet_manager.cc:1397] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.003s
I20250112 02:52:31.761950 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_feaebd2f4cbb47b6b24ff79614994d6f [id=63c20c66463d4a30b7e1558fecce2704]
I20250112 02:52:31.762151 1181 raft_consensus.cc:357] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.762365 1181 raft_consensus.cc:383] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.762396 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_5fb592e41cc4406aa27546e7b41fdb88 [id=6402abce8d1d45c182cc989490c6a5f0]
I20250112 02:52:31.762434 1181 raft_consensus.cc:738] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.762655 1181 consensus_queue.cc:260] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.763190 1181 ts_tablet_manager.cc:1428] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:31.762871 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]
I20250112 02:52:31.766942 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_9f651533b2a6469aa03a744720bdc14a [id=68e2f0649cb9417e89b8c6ba0d723d32]
I20250112 02:52:31.767782 1181 tablet_bootstrap.cc:492] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.769002 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_3e58cdf6ca254711b8071a2899d2402f [id=6a788a11657f4098bc906e16cd54618e]
I20250112 02:52:31.769405 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_9dd5993a01b44a0abe33a5738c8d5c2b [id=6a7dcf7429b24699b036c53b3005051f]
I20250112 02:52:31.769436 1181 tablet_bootstrap.cc:654] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.769876 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_26dfe346379948b39f90b31653c2a463 [id=6c6f30e60e79451397841d090d11d8b7]
I20250112 02:52:31.771744 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_7077f9a718664d8b915ee4003502306a [id=6df53b1d0c7a4fa1a18b3202741630b5]
I20250112 02:52:31.772583 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_c7107062f9574066bf9a4a362f9b3cfb [id=70e9b6c9e20f40eab186f042f21ad2ec]
I20250112 02:52:31.772955 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_569c0e413a014e52bcb98a9f4f644479 [id=75c94dcaccf344abbac5808bab28a673]
I20250112 02:52:31.773321 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_27f1bff714d947d5b3f42be6bfbe8be2 [id=78dff38cafdc48ddbc46b91a548c2022]
I20250112 02:52:31.773705 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]
I20250112 02:52:31.774596 1177 tablet_bootstrap.cc:654] T ed7243c6329d4f11aa25eea2b7b67683 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.774876 1609 raft_consensus.cc:491] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.775030 1609 raft_consensus.cc:513] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.775745 1609 leader_election.cc:290] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.777688 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8a178bafcb457f9b0e9587df560493" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.777622 1611 raft_consensus.cc:513] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.778015 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23e7a2e88c374e949ced92c557225cbf" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.778245 779 raft_consensus.cc:2388] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1e775296eb746db8b318f17d26820e6 in current term 1: Already voted for candidate fe36951a83fb4e57b26e187edbcbfe3d in this term.
I20250112 02:52:31.778386 1611 leader_election.cc:290] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.778661 991 leader_election.cc:304] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:31.779057 1634 raft_consensus.cc:3054] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.780395 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_d588e6b08cae48e590a4a57002df6323 [id=7a7301e1e2d74a6d9340ee996656f264]
I20250112 02:52:31.781841 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_525750dad2b7461586d070dbaa74d0b1 [id=7d048aa907a74605964b68726ac5d759]
I20250112 02:52:31.781893 1634 raft_consensus.cc:2743] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.782155 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61593969d2234ce3a6089027b067d922" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.782438 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_f89a1b732e34437294f2a7cdc052b6ea [id=7d79cefc22aa44309a4fed4d8198c79f]
I20250112 02:52:31.782440 1718 consensus_queue.cc:1035] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
I20250112 02:52:31.783013 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_9284616ca0cd499cb51df0322db9948f [id=7f36d8ba6d864d32ad84be06ca6682ee]
I20250112 02:52:31.783519 1173 tablet_bootstrap.cc:492] T 039a4734358646508093200d59d30013 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.783667 1173 ts_tablet_manager.cc:1397] T 039a4734358646508093200d59d30013 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.004s
I20250112 02:52:31.784404 1173 raft_consensus.cc:357] T 039a4734358646508093200d59d30013 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.784624 1173 raft_consensus.cc:383] T 039a4734358646508093200d59d30013 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.784715 1173 raft_consensus.cc:738] T 039a4734358646508093200d59d30013 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.784925 1173 consensus_queue.cc:260] T 039a4734358646508093200d59d30013 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.785162 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]
I20250112 02:52:31.785446 1173 ts_tablet_manager.cc:1428] T 039a4734358646508093200d59d30013 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:31.785737 1173 tablet_bootstrap.cc:492] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:31.785923 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_4603068c01544e2e965dca5177e36b40 [id=8502f21e9b61436093db1241cfbb5ec9]
W20250112 02:52:31.786471 855 leader_election.cc:343] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.786655 855 leader_election.cc:304] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.787045 1715 consensus_queue.cc:1035] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:31.787523 1715 raft_consensus.cc:2743] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.787633 1173 tablet_bootstrap.cc:654] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.789463 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161fe26b9c0e4543b7aeb0021506d72a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:31.789692 919 raft_consensus.cc:3054] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.793229 919 raft_consensus.cc:2463] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:31.793673 1714 raft_consensus.cc:491] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.793849 1714 raft_consensus.cc:513] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.794631 1714 leader_election.cc:290] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:31.795424 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a12b225131b464494a9f02133d37aea" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.795928 1181 tablet_bootstrap.cc:492] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.796130 1718 raft_consensus.cc:491] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.796248 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_4657847b84384e628194a8ff43e183a1 [id=85db347bdf4a4ff583062f3dddd20d8a]
I20250112 02:52:31.796310 1718 raft_consensus.cc:513] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.796424 1181 ts_tablet_manager.cc:1397] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.001s
I20250112 02:52:31.797052 1718 leader_election.cc:290] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.798120 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_365739770a11434dbf260ea912e7ea81 [id=868adb627654450c936ef198e49d853b]
I20250112 02:52:31.798406 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a12b225131b464494a9f02133d37aea" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:31.798655 780 raft_consensus.cc:2463] T 8a12b225131b464494a9f02133d37aea P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:31.799714 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]
I20250112 02:52:31.799491 1181 raft_consensus.cc:357] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.800012 1181 raft_consensus.cc:383] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 02:52:31.799428 854 leader_election.cc:343] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.800308 1181 raft_consensus.cc:738] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.801224 1181 consensus_queue.cc:260] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.802331 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_d45a3d1878fc47d5917bf66c670c5578 [id=8a4387d6a5084b85a8f28cb99798cec9]
I20250112 02:52:31.803303 1181 ts_tablet_manager.cc:1428] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250112 02:52:31.803972 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_29a6037a21724b5f9f0e0ee6a598a3dd [id=8b6988397e544a37b869f1e782095b64]
I20250112 02:52:31.804322 1173 tablet_bootstrap.cc:492] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.804487 1173 ts_tablet_manager.cc:1397] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.000s
I20250112 02:52:31.804700 1181 tablet_bootstrap.cc:492] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.805245 1173 raft_consensus.cc:357] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.805450 1173 raft_consensus.cc:383] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.805531 1173 raft_consensus.cc:738] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.805419 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_ad5f01f6a5274cc483c8f051059d36c3 [id=8be34ef1e0c54f7fa38ec57fb432b3db]
I20250112 02:52:31.805799 1173 consensus_queue.cc:260] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.806447 1173 ts_tablet_manager.cc:1428] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250112 02:52:31.806730 1173 tablet_bootstrap.cc:492] T 64707ee432fb4e96a3bbf7252dfa1813 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:31.807310 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_f8a40c27bebc483cbf62d5d0a4c7fb24 [id=8cc378b0da6e4b77bc2c6822cd0844fe]
I20250112 02:52:31.808359 1173 tablet_bootstrap.cc:654] T 64707ee432fb4e96a3bbf7252dfa1813 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.808769 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_4965c2235c4e422889169067ff632d29 [id=8dd987fcd63c4ee7a5fcabd32affaf3d]
I20250112 02:52:31.801285 855 leader_election.cc:304] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.809854 1714 raft_consensus.cc:2798] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.810024 1714 raft_consensus.cc:491] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.810173 1714 raft_consensus.cc:3054] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.811393 1181 tablet_bootstrap.cc:654] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.812014 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_c816af31a1e945518fd886cbf8b8b885 [id=93ab7e5945514c0aa85b411f15483f3d]
I20250112 02:52:31.812762 713 leader_election.cc:304] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:31.813295 1718 raft_consensus.cc:2798] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.813452 1718 raft_consensus.cc:695] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:31.813805 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_dc563f08a790498993b01afeec904975 [id=955093b0cd9a49bdbdc0517dc693a41d]
I20250112 02:52:31.813781 1718 consensus_queue.cc:237] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.816107 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161fe26b9c0e4543b7aeb0021506d72a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:31.816265 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c03b5e612d64314b210c11686fdfa2a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:31.816562 714 leader_election.cc:343] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.816318 1050 raft_consensus.cc:3054] T 161fe26b9c0e4543b7aeb0021506d72a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:31.822134 714 leader_election.cc:343] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.819427 1050 raft_consensus.cc:2463] T 161fe26b9c0e4543b7aeb0021506d72a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:31.833676 1634 consensus_queue.cc:1035] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:31.836634 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_63649e9607394f4bb72a3dd6e6d0dd69 [id=9807191b6e5c4d478cd47a54464fb9ce]
I20250112 02:52:31.837007 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c03b5e612d64314b210c11686fdfa2a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:31.838596 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_d3adef5e4ff541dabd0661d1c0997a71 [id=98b0d39d324c4b2b8e83dbb31776a5cd]
I20250112 02:52:31.843482 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_6d12502c6e714726b9d807690fd21ea4 [id=9b5607155d264cf2879d18305398df11]
I20250112 02:52:31.843802 1173 tablet_bootstrap.cc:492] T 64707ee432fb4e96a3bbf7252dfa1813 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.843972 1173 ts_tablet_manager.cc:1397] T 64707ee432fb4e96a3bbf7252dfa1813 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.015s
I20250112 02:52:31.845430 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_ad69cfacaf7b4050ab2d3d6ed237e7d0 [id=9e72fad141db43c09c7e4692769b6eba]
I20250112 02:52:31.845641 1173 raft_consensus.cc:357] T 64707ee432fb4e96a3bbf7252dfa1813 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.845863 1173 raft_consensus.cc:383] T 64707ee432fb4e96a3bbf7252dfa1813 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.845949 1173 raft_consensus.cc:738] T 64707ee432fb4e96a3bbf7252dfa1813 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.846189 1173 consensus_queue.cc:260] T 64707ee432fb4e96a3bbf7252dfa1813 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.848655 1173 ts_tablet_manager.cc:1428] T 64707ee432fb4e96a3bbf7252dfa1813 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250112 02:52:31.848798 1634 consensus_queue.cc:1035] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:31.848982 1173 tablet_bootstrap.cc:492] T abfaad67d5e34c0a8338dccc0ef05466 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:31.850831 1173 tablet_bootstrap.cc:654] T abfaad67d5e34c0a8338dccc0ef05466 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.855046 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_5e6ad2a2040d41b381bb2653c6f691ec [id=a2024dc6a87341cb87c45eee7ab024b0]
I20250112 02:52:31.856053 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_6b8cb70a5e5b4519af5d247f6b5bb8b0 [id=a28f5cfbcfe64768b40477ffc2053e27]
I20250112 02:52:31.856843 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_ab4d7c58924a4b8682b52637c23f125b [id=a622f7a973344a388b3275adbcce6234]
I20250112 02:52:31.858098 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_63abe3ac694a4ed6bf4aef720b542e73 [id=a7ce19c248fe4ac6a6b436773d6c1b4c]
I20250112 02:52:31.861068 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]
I20250112 02:52:31.863170 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_b0af74e9ea8a4c5faea64696a16d5a9a [id=ad0be37e6b6f4b9597dcb4cbd50d5b12]
I20250112 02:52:31.863869 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]
I20250112 02:52:31.864985 1173 tablet_bootstrap.cc:492] T abfaad67d5e34c0a8338dccc0ef05466 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:31.865131 1173 ts_tablet_manager.cc:1397] T abfaad67d5e34c0a8338dccc0ef05466 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20250112 02:52:31.865638 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_76e4d0c54155412296bc900e03cd40bd [id=aecf277dcd794bf3bf27d58c9b96dd12]
I20250112 02:52:31.865823 1173 raft_consensus.cc:357] T abfaad67d5e34c0a8338dccc0ef05466 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.866039 1173 raft_consensus.cc:383] T abfaad67d5e34c0a8338dccc0ef05466 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.866099 1173 raft_consensus.cc:738] T abfaad67d5e34c0a8338dccc0ef05466 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.866309 1173 consensus_queue.cc:260] T abfaad67d5e34c0a8338dccc0ef05466 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.866876 1173 ts_tablet_manager.cc:1428] T abfaad67d5e34c0a8338dccc0ef05466 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:31.867636 1173 tablet_bootstrap.cc:492] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:31.868294 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]
I20250112 02:52:31.869796 919 raft_consensus.cc:2463] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:31.870230 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_743b3f7029534564b35c0c137b35fd75 [id=b2fa3922f6d447769a8179052f9a1861]
I20250112 02:52:31.871137 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_f7421d0779924239a3b493954ef748b7 [id=b42fa8fc65e344f989a98ae411e916ed]
I20250112 02:52:31.871858 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_1216acc1fa6b4e51a55a89ffe22bcc9a [id=b5457e4343ee43618c8309e737b69ec3]
I20250112 02:52:31.872740 713 leader_election.cc:304] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.873029 1173 tablet_bootstrap.cc:654] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.873023 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_fc264b6a06584c7ea757f5d18e05e2bc [id=b560a3488e3d4db58abc7d81d6094d86]
I20250112 02:52:31.873322 1602 raft_consensus.cc:2798] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.873412 1602 raft_consensus.cc:491] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.873497 1602 raft_consensus.cc:3054] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.875595 1741 consensus_queue.cc:1035] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:31.869890 1715 consensus_queue.cc:1035] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20250112 02:52:31.878218 1602 raft_consensus.cc:513] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.879133 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]
I20250112 02:52:31.879168 1602 leader_election.cc:290] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.879618 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c03b5e612d64314b210c11686fdfa2a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:31.880508 714 leader_election.cc:343] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.867319 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8a178bafcb457f9b0e9587df560493" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:31.882076 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_6f3879f2469d4655a6d4436362d0c3bb [id=bb0ba72d0ee747a3945283096d717008]
I20250112 02:52:31.882827 1602 consensus_queue.cc:1035] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:31.891680 916 raft_consensus.cc:2463] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:31.895301 713 leader_election.cc:304] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.889770 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c03b5e612d64314b210c11686fdfa2a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:31.898039 1602 raft_consensus.cc:2798] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.898042 1634 consensus_queue.cc:1035] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:31.898159 1602 raft_consensus.cc:491] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.898361 1602 raft_consensus.cc:3054] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.903949 918 raft_consensus.cc:3054] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.911780 1634 consensus_queue.cc:1035] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:31.915495 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_cd1f185d82b3474083246ec3702851fc [id=bfa9a504a8d34c86929efb0d15d049d0]
I20250112 02:52:31.920219 918 raft_consensus.cc:2463] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:31.920724 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_7926c082d6c04315811ea3b4e1579e8c [id=c01b653cd9fe45b2af721eb97936c5d4]
I20250112 02:52:31.920697 1602 raft_consensus.cc:513] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.921623 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_9c10574f7c7d435890eb9a24af23bcb9 [id=c0e16b77525146a28b8182137bc81566]
I20250112 02:52:31.923189 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8a178bafcb457f9b0e9587df560493" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:31.923823 714 leader_election.cc:343] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.925483 1602 leader_election.cc:290] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.926165 1714 raft_consensus.cc:513] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.927132 1714 leader_election.cc:290] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:31.927454 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a12b225131b464494a9f02133d37aea" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:31.927803 774 raft_consensus.cc:3054] T 8a12b225131b464494a9f02133d37aea P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.930390 713 leader_election.cc:304] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.930655 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8a178bafcb457f9b0e9587df560493" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:31.930785 1602 raft_consensus.cc:2798] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.930913 774 raft_consensus.cc:2463] T 8a12b225131b464494a9f02133d37aea P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:31.931048 1602 raft_consensus.cc:695] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:31.931319 1602 consensus_queue.cc:237] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.931538 855 leader_election.cc:304] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:31.931926 1714 raft_consensus.cc:491] T ed7243c6329d4f11aa25eea2b7b67683 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.933521 1859 raft_consensus.cc:491] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.933938 1859 raft_consensus.cc:513] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:31.935359 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a18132549374218893f35a4e4226b72" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:31.935886 855 leader_election.cc:343] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.936252 1859 leader_election.cc:290] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:31.937561 1859 raft_consensus.cc:2798] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.937809 1859 raft_consensus.cc:695] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:31.938180 1859 consensus_queue.cc:237] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.940430 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]
I20250112 02:52:31.943688 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_009a6976f6db4f218e3c3373368a58d7 [id=c5005669bb5f4c029d673bd39cc1535d]
I20250112 02:52:31.947301 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_69f363c4da234792886b544823b3214e [id=c7052afaf6284ef0972835c1f46cb829]
I20250112 02:52:31.950207 1714 raft_consensus.cc:513] T ed7243c6329d4f11aa25eea2b7b67683 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.950732 1181 tablet_bootstrap.cc:492] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:31.952384 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_b8af9fadb7b64c07afb49c07cf3ca5f9 [id=c7f00d3d40fb4a469c57e3f82983ff53]
I20250112 02:52:31.957508 917 raft_consensus.cc:3054] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.957454 1181 ts_tablet_manager.cc:1397] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.146s user 0.007s sys 0.007s
I20250112 02:52:31.958604 1718 raft_consensus.cc:491] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.961068 1714 leader_election.cc:290] T ed7243c6329d4f11aa25eea2b7b67683 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:31.961397 1714 raft_consensus.cc:491] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.964890 917 raft_consensus.cc:2463] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:31.965448 713 leader_election.cc:304] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.965816 1741 raft_consensus.cc:2798] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.965960 1741 raft_consensus.cc:695] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:31.965916 1181 raft_consensus.cc:357] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.966243 1741 consensus_queue.cc:237] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:31.966296 1857 consensus_queue.cc:1035] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250112 02:52:31.966750 1857 raft_consensus.cc:491] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.966876 1857 raft_consensus.cc:513] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.967468 1857 leader_election.cc:290] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:31.967782 1181 raft_consensus.cc:383] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:31.967891 1181 raft_consensus.cc:738] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:31.968171 1181 consensus_queue.cc:260] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.968834 1181 ts_tablet_manager.cc:1428] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20250112 02:52:31.969368 1181 tablet_bootstrap.cc:492] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:31.971909 1714 raft_consensus.cc:513] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.972486 1714 leader_election.cc:290] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:31.974074 1718 raft_consensus.cc:513] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.975559 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_15bcb2c0f2a74458a1bed231fd3437fc [id=ca9c8281bf254f248076cb600e7c4f87]
I20250112 02:52:31.976990 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a18132549374218893f35a4e4226b72" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.977267 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7243c6329d4f11aa25eea2b7b67683" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.977564 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a12b225131b464494a9f02133d37aea" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:31.978955 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccff64af8ca4400afc116db98918119" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.979671 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015412a5fa304453bd8d1be1c3cd7a46" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.979866 1054 raft_consensus.cc:2463] T 015412a5fa304453bd8d1be1c3cd7a46 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:31.980259 714 leader_election.cc:304] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:31.980541 1609 raft_consensus.cc:2798] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:31.980562 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73136be25c79491784d3ea71aa9b27f2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:31.980675 1609 raft_consensus.cc:491] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:31.980767 1609 raft_consensus.cc:3054] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.981787 1181 tablet_bootstrap.cc:654] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:31.981513 1718 leader_election.cc:290] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.981726 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_c26869f6cbd540fe980dcf207b5c729d [id=cdd6a2097792484d84bca856f30d7ac1]
I20250112 02:52:31.983089 1054 raft_consensus.cc:3054] T 64707ee432fb4e96a3bbf7252dfa1813 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.983204 1609 raft_consensus.cc:513] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.984163 1609 leader_election.cc:290] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:31.984638 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015412a5fa304453bd8d1be1c3cd7a46" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:31.984810 1051 raft_consensus.cc:3054] T 015412a5fa304453bd8d1be1c3cd7a46 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:31.986678 989 consensus_peers.cc:487] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65f22a7b1b804661af3b7e785a833739. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:31.987682 1051 raft_consensus.cc:2463] T 015412a5fa304453bd8d1be1c3cd7a46 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
W20250112 02:52:31.987840 854 leader_election.cc:343] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:31.988072 714 leader_election.cc:304] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:31.988510 1609 raft_consensus.cc:2798] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:31.988667 1609 raft_consensus.cc:695] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:31.988186 854 leader_election.cc:304] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:31.988922 1609 consensus_queue.cc:237] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.989477 1857 raft_consensus.cc:2743] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:31.990103 1857 raft_consensus.cc:491] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.990397 1857 raft_consensus.cc:513] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:31.990523 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015412a5fa304453bd8d1be1c3cd7a46" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:31.990885 918 raft_consensus.cc:3054] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:31.993438 918 raft_consensus.cc:2463] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:31.994225 1857 leader_election.cc:290] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:31.999593 1533 raft_consensus.cc:491] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:31.999920 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015412a5fa304453bd8d1be1c3cd7a46" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:32.000167 917 raft_consensus.cc:2371] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1. Re-sending same reply.
I20250112 02:52:32.000285 1533 raft_consensus.cc:513] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.005017 1602 consensus_queue.cc:1035] T 27ab58e45d0d44b283019240f44b66bf P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.005923 1533 leader_election.cc:290] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
W20250112 02:52:32.006224 854 leader_election.cc:343] T ed7243c6329d4f11aa25eea2b7b67683 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.007117 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_f3695b00a89b4353903587a6c1418d8c [id=d1781738b8a74f139b09863918cbb21d]
W20250112 02:52:32.007429 854 leader_election.cc:343] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:32.009665 854 leader_election.cc:343] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:32.015323 854 leader_election.cc:343] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.015525 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]
I20250112 02:52:32.015980 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b70dce4d1d4e29997a18297ac0f82a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.017627 1533 consensus_queue.cc:1035] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.017861 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b70dce4d1d4e29997a18297ac0f82a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:32.018394 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccff64af8ca4400afc116db98918119" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.019104 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1811ce8982fd4003a35e89ea34e32746" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.019601 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1811ce8982fd4003a35e89ea34e32746" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:32.020092 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73136be25c79491784d3ea71aa9b27f2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.021173 1173 tablet_bootstrap.cc:492] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:32.021381 1173 ts_tablet_manager.cc:1397] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.154s user 0.002s sys 0.011s
I20250112 02:52:32.022158 1173 raft_consensus.cc:357] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.022398 1173 raft_consensus.cc:383] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.022538 1173 raft_consensus.cc:738] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.023546 1173 consensus_queue.cc:260] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.023444 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_178713b464bb4f16970851646a710574 [id=d3f31247f0e848f8b2dfa18f3437ca63]
I20250112 02:52:32.023952 1871 raft_consensus.cc:491] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.024068 1871 raft_consensus.cc:513] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:32.024652 854 leader_election.cc:343] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:32.025821 855 leader_election.cc:343] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.025980 855 leader_election.cc:304] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:32.026405 1857 raft_consensus.cc:2743] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.024755 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7243c6329d4f11aa25eea2b7b67683" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.027226 1181 tablet_bootstrap.cc:492] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:32.027207 1871 leader_election.cc:290] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:32.027372 1181 ts_tablet_manager.cc:1397] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.058s user 0.001s sys 0.003s
I20250112 02:52:32.028090 1181 raft_consensus.cc:357] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.028275 1181 raft_consensus.cc:383] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.028353 1181 raft_consensus.cc:738] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.028566 1181 consensus_queue.cc:260] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.029073 1181 ts_tablet_manager.cc:1428] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:32.029183 1050 raft_consensus.cc:3054] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.029323 1181 tablet_bootstrap.cc:492] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:32.031030 1181 tablet_bootstrap.cc:654] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:32.026372 854 leader_election.cc:343] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:32.032123 855 leader_election.cc:343] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.032300 855 leader_election.cc:304] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:32.032799 855 leader_election.cc:343] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.032956 855 leader_election.cc:304] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:32.033545 1857 raft_consensus.cc:2743] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.033800 1857 raft_consensus.cc:2743] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.033872 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]
I20250112 02:52:32.035953 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c01b80fc8242d98fdfbf767212144d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:32.036823 855 leader_election.cc:343] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.037298 855 leader_election.cc:304] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:32.037568 1533 raft_consensus.cc:2743] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.038326 1611 consensus_queue.cc:1035] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.024571 1173 ts_tablet_manager.cc:1428] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
W20250112 02:52:32.040437 855 leader_election.cc:343] T ed7243c6329d4f11aa25eea2b7b67683 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.040459 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_e80769df10444dd5ae4bc23a8c6cc09e [id=d91a9f9e1f76482e99d4f10c0568f2e3]
I20250112 02:52:32.040664 1173 tablet_bootstrap.cc:492] T 9f0314aa03b1473ebbe05ce73adf7838 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:32.041033 855 leader_election.cc:304] T ed7243c6329d4f11aa25eea2b7b67683 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:32.041615 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_0e7ff0c4e04e4b23a8baf069f96ac465 [id=d98e9e3bcb104f48a5102dea02c73c95]
I20250112 02:52:32.042164 1861 raft_consensus.cc:2743] T ed7243c6329d4f11aa25eea2b7b67683 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.043581 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_d561e1740cc545dc8aa4babb76b45c37 [id=de27aeff6b3a49b3956498ea5015ff33]
I20250112 02:52:32.045126 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_f173732a21194c7aa8f9a40eb7eb07a1 [id=e04012ba08a640828b73c52803bf214b]
I20250112 02:52:32.045985 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_6671f628f89b428a83eca77e9ca26516 [id=e0577e1a40064437bb7eb4c66fbd914e]
I20250112 02:52:32.048981 1851 raft_consensus.cc:491] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.049153 1851 raft_consensus.cc:513] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.049767 1851 leader_election.cc:290] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.050371 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d659c5a4f7a446c6aaf2283f65b45106" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.051079 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_096703f878244b8dac2dcf187fd23fd6 [id=e1779f88b20942eaa2c640ac24946fff]
I20250112 02:52:32.052393 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_46a03fa2d75e4541a98a1b3d91f85395 [id=e184c10af82240f5b1e29fc226269bf0]
W20250112 02:52:32.053267 855 leader_election.cc:343] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.053828 1173 tablet_bootstrap.cc:654] T 9f0314aa03b1473ebbe05ce73adf7838 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.055663 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_2b5aeec3b0bf40aea08010f65e422c2c [id=e292141365e9453fa1816e0f313b2300]
I20250112 02:52:32.057008 1851 raft_consensus.cc:491] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.057211 1851 raft_consensus.cc:513] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.057823 1851 leader_election.cc:290] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.058998 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48634d6d78784fae98512b242e5476df" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.059195 1734 consensus_queue.cc:1035] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:52:32.059536 855 leader_election.cc:343] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.062503 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]
I20250112 02:52:32.065991 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_0bdc5f49d4ff4d6fb1290bc2dc0f70d9 [id=e3bb489c53454222a79702b798c0c2f1]
I20250112 02:52:32.066993 1173 tablet_bootstrap.cc:492] T 9f0314aa03b1473ebbe05ce73adf7838 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:32.068776 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]
I20250112 02:52:32.069433 1173 ts_tablet_manager.cc:1397] T 9f0314aa03b1473ebbe05ce73adf7838 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.003s
I20250112 02:52:32.070739 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_9b65dd291c9f4fcdaa6d5e37f04d24ae [id=e63daf724d874c89badafb8d59653f74]
I20250112 02:52:32.071784 1173 raft_consensus.cc:357] T 9f0314aa03b1473ebbe05ce73adf7838 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.072012 1173 raft_consensus.cc:383] T 9f0314aa03b1473ebbe05ce73adf7838 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.072088 1173 raft_consensus.cc:738] T 9f0314aa03b1473ebbe05ce73adf7838 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.072278 1173 consensus_queue.cc:260] T 9f0314aa03b1473ebbe05ce73adf7838 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.073529 1181 tablet_bootstrap.cc:492] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:32.075706 1181 ts_tablet_manager.cc:1397] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.008s
I20250112 02:52:32.076295 1173 ts_tablet_manager.cc:1428] T 9f0314aa03b1473ebbe05ce73adf7838 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250112 02:52:32.076588 1173 tablet_bootstrap.cc:492] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:32.081650 1173 tablet_bootstrap.cc:654] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.082998 1181 raft_consensus.cc:357] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.083745 1181 raft_consensus.cc:383] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.083969 1181 raft_consensus.cc:738] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.086035 1634 raft_consensus.cc:491] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.084836 1181 consensus_queue.cc:260] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.086273 1634 raft_consensus.cc:513] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.086838 1181 ts_tablet_manager.cc:1428] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20250112 02:52:32.087260 1634 leader_election.cc:290] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:32.088366 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cb7e7a41634cf1bdb978ac9d2befcd" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.088614 779 raft_consensus.cc:2463] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:32.088935 1181 tablet_bootstrap.cc:492] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:32.089004 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_c11c44af47604f9589b15d5c6a97c255 [id=e80fb97e841a4e68a6ab992d897816c1]
I20250112 02:52:32.089428 1861 raft_consensus.cc:491] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.089687 1861 raft_consensus.cc:513] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.091243 1861 leader_election.cc:290] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.092206 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1e3af2883341e9b957f6ed748e604d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.092530 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_31424b670f274fd3abd17eb8451ba95c [id=ebc9aae273fb4bcd8bd904f2179f1dd4]
I20250112 02:52:32.092844 1173 tablet_bootstrap.cc:492] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
W20250112 02:52:32.092774 855 leader_election.cc:343] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.092983 1173 ts_tablet_manager.cc:1397] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250112 02:52:32.093277 991 leader_election.cc:304] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:32.093569 1634 raft_consensus.cc:2798] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:32.093713 1634 raft_consensus.cc:491] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:32.093915 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_add1776e850b4a1bbd63f05707b51873 [id=ec37a56b10ba407b86e851f07bddf528]
I20250112 02:52:32.094118 1634 raft_consensus.cc:3054] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.094324 1718 raft_consensus.cc:491] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.094326 1050 raft_consensus.cc:3054] T 1c639a2bcd4740e4be7b2b162004669a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.094477 1718 raft_consensus.cc:513] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.095671 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea075826b16475ca474e284fc6bc545" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:32.096282 714 leader_election.cc:343] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.097223 1634 raft_consensus.cc:513] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.097555 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea075826b16475ca474e284fc6bc545" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:32.097731 917 raft_consensus.cc:2463] T 7ea075826b16475ca474e284fc6bc545 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:32.098024 1634 leader_election.cc:290] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:32.098079 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cb7e7a41634cf1bdb978ac9d2befcd" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:32.098547 779 raft_consensus.cc:3054] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.098384 713 leader_election.cc:304] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:32.099112 1181 tablet_bootstrap.cc:654] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.100019 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_952b91d2b0b8435886e555f12629dc80 [id=ed2c162c64f74c9f889789b3abe7c041]
I20250112 02:52:32.100337 1173 raft_consensus.cc:357] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.100528 1173 raft_consensus.cc:383] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.100641 1173 raft_consensus.cc:738] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.102173 779 raft_consensus.cc:2463] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:32.102489 1609 consensus_queue.cc:1035] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.102964 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_f834660b442848b3b36f079110711bb3 [id=f32f4eea5a904798baa70ed435c9bf62]
I20250112 02:52:32.103863 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_e495ee84abe3479c80c8ab2e14f86c96 [id=f5eb0a253e444d76978a22250c163a54]
I20250112 02:52:32.103906 1609 raft_consensus.cc:2798] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:32.104014 1609 raft_consensus.cc:491] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:32.104154 1609 raft_consensus.cc:3054] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.107178 1609 raft_consensus.cc:513] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.107975 991 leader_election.cc:304] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:32.108379 1876 raft_consensus.cc:2798] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:32.108517 1876 raft_consensus.cc:695] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:32.108788 1876 consensus_queue.cc:237] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.112035 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea075826b16475ca474e284fc6bc545" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:32.112572 714 leader_election.cc:343] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.113559 1054 raft_consensus.cc:3054] T e3ee5d8a17874d7ba54de51568e7d2e6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.117182 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]
I20250112 02:52:32.117681 1734 consensus_queue.cc:1035] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.119772 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_7ee8a748c60d439088423dcb284b17ad [id=f836cdee435949d387336c9a52b6ed1b]
I20250112 02:52:32.122670 1609 leader_election.cc:290] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:32.097738 1718 leader_election.cc:290] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:32.126479 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea075826b16475ca474e284fc6bc545" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:32.126693 917 raft_consensus.cc:3054] T 7ea075826b16475ca474e284fc6bc545 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.128348 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c01b80fc8242d98fdfbf767212144d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:32.129381 917 raft_consensus.cc:2463] T 7ea075826b16475ca474e284fc6bc545 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:32.130259 713 leader_election.cc:304] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:32.133358 1741 consensus_queue.cc:1035] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.133695 1611 consensus_queue.cc:1035] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.134142 1611 raft_consensus.cc:2798] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:32.134270 1611 raft_consensus.cc:695] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:32.134532 1611 consensus_queue.cc:237] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.135352 1173 consensus_queue.cc:260] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.136050 1173 ts_tablet_manager.cc:1428] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.043s user 0.000s sys 0.002s
I20250112 02:52:32.136955 1173 tablet_bootstrap.cc:492] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:32.138921 1173 tablet_bootstrap.cc:654] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.139245 1734 consensus_queue.cc:1035] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.148703 1181 tablet_bootstrap.cc:492] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:32.149467 1181 ts_tablet_manager.cc:1397] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.002s
I20250112 02:52:32.150682 1861 raft_consensus.cc:491] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.156383 1173 tablet_bootstrap.cc:492] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:32.156538 1173 ts_tablet_manager.cc:1397] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.003s
I20250112 02:52:32.156293 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_3321055e22c3410eb5e5fe8be62e004c [id=f8cd4289f89a4a748c26a7a01236406e]
I20250112 02:52:32.157402 1173 raft_consensus.cc:357] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.157608 1173 raft_consensus.cc:383] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.157687 1173 raft_consensus.cc:738] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.167078 1861 raft_consensus.cc:513] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.168460 1173 consensus_queue.cc:260] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.169689 1851 raft_consensus.cc:491] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.169871 1851 raft_consensus.cc:513] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.170298 1173 ts_tablet_manager.cc:1428] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.014s user 0.000s sys 0.003s
I20250112 02:52:32.170606 1173 tablet_bootstrap.cc:492] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:32.172212 1173 tablet_bootstrap.cc:654] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:32.173775 989 consensus_peers.cc:487] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9163878cfeb64693a9c5493e812546af. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:32.174230 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d69632845547f588662dd56c8c194e" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:32.174526 989 consensus_peers.cc:487] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c686fdc8896463c81c9f7c5f53a3da2. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:52:32.175300 989 consensus_peers.cc:487] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c0c2fad4f3243238641aae56b305bb5. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:32.177322 989 leader_election.cc:304] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
W20250112 02:52:32.180020 989 leader_election.cc:343] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 42cb7e7a41634cf1bdb978ac9d2befcd
W20250112 02:52:32.181859 989 leader_election.cc:343] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 42cb7e7a41634cf1bdb978ac9d2befcd
I20250112 02:52:32.182266 1851 leader_election.cc:290] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
W20250112 02:52:32.182610 855 leader_election.cc:343] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.180294 1875 raft_consensus.cc:2743] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.183172 1851 raft_consensus.cc:491] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.183359 1851 raft_consensus.cc:513] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.184419 1851 leader_election.cc:290] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.185004 1861 leader_election.cc:290] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.185513 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb217afc08304c0ea35dee3120347ba1" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:32.186031 855 leader_election.cc:343] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.186075 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5b3df88a0c45dba879b3d506710614" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:32.187121 855 leader_election.cc:343] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.187988 1177 tablet_bootstrap.cc:492] T ed7243c6329d4f11aa25eea2b7b67683 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:32.188222 1177 ts_tablet_manager.cc:1397] T ed7243c6329d4f11aa25eea2b7b67683 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.493s user 0.063s sys 0.131s
I20250112 02:52:32.187951 1181 raft_consensus.cc:357] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:32.188742 1181 raft_consensus.cc:383] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.189000 1181 raft_consensus.cc:738] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.189203 1177 raft_consensus.cc:357] T ed7243c6329d4f11aa25eea2b7b67683 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.189529 1177 raft_consensus.cc:383] T ed7243c6329d4f11aa25eea2b7b67683 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.189635 1177 raft_consensus.cc:738] T ed7243c6329d4f11aa25eea2b7b67683 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.189370 1181 consensus_queue.cc:260] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:32.189950 1177 consensus_queue.cc:260] T ed7243c6329d4f11aa25eea2b7b67683 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.192131 1609 raft_consensus.cc:491] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.192318 1609 raft_consensus.cc:513] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.192955 1609 leader_election.cc:290] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:32.193169 1734 consensus_queue.cc:1035] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.193264 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0314aa03b1473ebbe05ce73adf7838" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:32.193449 1050 raft_consensus.cc:2463] T 9f0314aa03b1473ebbe05ce73adf7838 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:32.193989 714 leader_election.cc:304] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:32.194703 1050 raft_consensus.cc:3054] T abfaad67d5e34c0a8338dccc0ef05466 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.197165 1177 ts_tablet_manager.cc:1428] T ed7243c6329d4f11aa25eea2b7b67683 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250112 02:52:32.197482 1177 tablet_bootstrap.cc:492] T 49bb6d20cfa0453e90e048825114525e P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:32.198380 1734 raft_consensus.cc:2798] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:32.198508 1734 raft_consensus.cc:491] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:32.198611 1734 raft_consensus.cc:3054] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.200342 1177 tablet_bootstrap.cc:654] T 49bb6d20cfa0453e90e048825114525e P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.201480 1734 raft_consensus.cc:513] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.202929 1734 leader_election.cc:290] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:32.203258 1734 consensus_queue.cc:1035] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.204702 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0314aa03b1473ebbe05ce73adf7838" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:32.204986 1054 raft_consensus.cc:3054] T 9f0314aa03b1473ebbe05ce73adf7838 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.205988 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0314aa03b1473ebbe05ce73adf7838" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:32.206264 917 raft_consensus.cc:3054] T 9f0314aa03b1473ebbe05ce73adf7838 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.208096 1173 tablet_bootstrap.cc:492] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:32.208225 1173 ts_tablet_manager.cc:1397] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.003s
I20250112 02:52:32.208699 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0314aa03b1473ebbe05ce73adf7838" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:32.209117 1173 raft_consensus.cc:357] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.209775 1054 raft_consensus.cc:2463] T 9f0314aa03b1473ebbe05ce73adf7838 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:32.209961 1173 raft_consensus.cc:383] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.211094 1173 raft_consensus.cc:738] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.211356 1173 consensus_queue.cc:260] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.211525 714 leader_election.cc:304] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:32.211931 1734 raft_consensus.cc:2798] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:32.212118 1734 raft_consensus.cc:695] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:32.212188 1173 ts_tablet_manager.cc:1428] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:32.212416 1734 consensus_queue.cc:237] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.215555 917 raft_consensus.cc:2463] T 9f0314aa03b1473ebbe05ce73adf7838 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:32.217298 1173 tablet_bootstrap.cc:492] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:32.192530 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_613140584042445b9d71a3d05cee62c1 [id=f95ac050a4b840a19a8175caade87d48]
I20250112 02:52:32.221194 1173 tablet_bootstrap.cc:654] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.221097 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_6ff2131cc2144af7b6c223f14e9eb25a [id=fc8a29e5f2e24a4784009e85a98a40bd]
I20250112 02:52:32.217128 1851 raft_consensus.cc:491] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.226673 1851 raft_consensus.cc:513] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.227484 1851 leader_election.cc:290] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.227870 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79c6bdd89f5a44efb56d65ccb724308b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:32.228358 855 leader_election.cc:343] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.192648 1181 ts_tablet_manager.cc:1428] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.042s user 0.001s sys 0.002s
I20250112 02:52:32.229733 1181 tablet_bootstrap.cc:492] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:32.233537 1875 raft_consensus.cc:491] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.233757 1875 raft_consensus.cc:513] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.234522 1875 leader_election.cc:290] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.235024 1181 tablet_bootstrap.cc:654] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.238114 1706 catalog_manager.cc:670] Loaded metadata for table default.Table_977c68d65cd443ef936f3dd77b445154 [id=fee2f9e65e314ce9a1fc1fb6dfcde317]
I20250112 02:52:32.244626 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8a178bafcb457f9b0e9587df560493" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.249405 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48634d6d78784fae98512b242e5476df" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:32.249770 1177 tablet_bootstrap.cc:492] T 49bb6d20cfa0453e90e048825114525e P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:32.249737 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d659c5a4f7a446c6aaf2283f65b45106" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:32.249964 1177 ts_tablet_manager.cc:1397] T 49bb6d20cfa0453e90e048825114525e P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.014s
I20250112 02:52:32.250213 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb217afc08304c0ea35dee3120347ba1" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:32.250844 1177 raft_consensus.cc:357] T 49bb6d20cfa0453e90e048825114525e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.251062 1177 raft_consensus.cc:383] T 49bb6d20cfa0453e90e048825114525e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.251196 1177 raft_consensus.cc:738] T 49bb6d20cfa0453e90e048825114525e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.251518 1177 consensus_queue.cc:260] T 49bb6d20cfa0453e90e048825114525e P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.252177 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d69632845547f588662dd56c8c194e" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:32.251833 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1e3af2883341e9b957f6ed748e604d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:32.253655 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5b3df88a0c45dba879b3d506710614" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:32.254643 854 leader_election.cc:343] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.254791 854 leader_election.cc:304] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:32.255183 854 leader_election.cc:343] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.255306 854 leader_election.cc:304] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:32.255658 854 leader_election.cc:343] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.255749 854 leader_election.cc:304] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:32.256106 854 leader_election.cc:343] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.256194 854 leader_election.cc:304] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:32.256554 854 leader_election.cc:343] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.256645 854 leader_election.cc:304] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:32.256986 854 leader_election.cc:343] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.257097 854 leader_election.cc:304] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:32.257782 1851 raft_consensus.cc:2743] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.257966 1851 raft_consensus.cc:2743] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.258109 1851 raft_consensus.cc:2743] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.258481 1851 raft_consensus.cc:2743] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.258603 1851 raft_consensus.cc:2743] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.258745 1851 raft_consensus.cc:2743] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.259104 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8a178bafcb457f9b0e9587df560493" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:32.259297 917 raft_consensus.cc:2388] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1e775296eb746db8b318f17d26820e6 in current term 1: Already voted for candidate 289afc116aa74408aa5bff74e11c50d8 in this term.
I20250112 02:52:32.260335 1177 ts_tablet_manager.cc:1428] T 49bb6d20cfa0453e90e048825114525e P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20250112 02:52:32.260861 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79c6bdd89f5a44efb56d65ccb724308b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:32.260870 1877 raft_consensus.cc:491] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.261329 1877 raft_consensus.cc:513] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.262949 1875 consensus_queue.cc:1035] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250112 02:52:32.263156 1173 tablet_bootstrap.cc:492] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:32.263355 1859 raft_consensus.cc:491] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.263479 1859 raft_consensus.cc:513] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.263988 1859 leader_election.cc:290] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
W20250112 02:52:32.265388 854 leader_election.cc:343] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.265733 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd48d0e68014e1aa0c44620527baec9" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.269186 1173 ts_tablet_manager.cc:1397] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.007s
I20250112 02:52:32.269632 1866 raft_consensus.cc:491] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.269789 1866 raft_consensus.cc:513] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:32.270418 1866 leader_election.cc:290] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.270748 1877 leader_election.cc:290] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:32.270897 1054 raft_consensus.cc:3054] T 7cd9985cd20148f195a857c478b0755f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:32.266177 855 leader_election.cc:343] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.265496 854 leader_election.cc:304] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:32.271697 1857 raft_consensus.cc:2743] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.272064 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6913b0ac9ceb4dbc9cdf591f159134f7" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:32.272421 1173 raft_consensus.cc:357] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.273926 989 leader_election.cc:304] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:32.275621 1877 raft_consensus.cc:3054] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.272635 1173 raft_consensus.cc:383] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.276446 1173 raft_consensus.cc:738] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.276727 1173 consensus_queue.cc:260] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.277370 1857 raft_consensus.cc:491] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.277500 1857 raft_consensus.cc:513] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.278076 1857 leader_election.cc:290] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.277308 1859 consensus_queue.cc:1035] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.278169 1173 ts_tablet_manager.cc:1428] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250112 02:52:32.278606 1173 tablet_bootstrap.cc:492] T 2c03b5e612d64314b210c11686fdfa2a P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:32.283087 989 leader_election.cc:343] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: c7c530aa0aa64e47ae27d61f341d2bd9
W20250112 02:52:32.283336 989 consensus_peers.cc:487] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35f59617b34046568dd83bbb7cdc42d2. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:32.287178 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b599e983ec04f53a440d1ab828d12ab" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:32.287588 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd48d0e68014e1aa0c44620527baec9" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:32.288115 854 leader_election.cc:343] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:32.288352 854 leader_election.cc:343] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.288686 854 leader_election.cc:304] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:32.288913 1859 raft_consensus.cc:2743] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.289371 1877 raft_consensus.cc:2743] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.290836 1173 tablet_bootstrap.cc:654] T 2c03b5e612d64314b210c11686fdfa2a P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.291754 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6913b0ac9ceb4dbc9cdf591f159134f7" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.298689 1877 raft_consensus.cc:491] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.298722 1706 tablet_loader.cc:96] loaded metadata for tablet 003a3b4705e943c6a900762ec11ddd47 (table default.Table_743b3f7029534564b35c0c137b35fd75 [id=b2fa3922f6d447769a8179052f9a1861])
I20250112 02:52:32.298928 1877 raft_consensus.cc:513] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.299870 1706 tablet_loader.cc:96] loaded metadata for tablet 039a4734358646508093200d59d30013 (table default.Table_f834660b442848b3b36f079110711bb3 [id=f32f4eea5a904798baa70ed435c9bf62])
I20250112 02:52:32.300220 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d0f67141dc46b6a87ca0ad080d5bfe" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:32.300315 1706 tablet_loader.cc:96] loaded metadata for tablet 03a32b424ec4431d88f6af45dd09089f (table default.Table_f173732a21194c7aa8f9a40eb7eb07a1 [id=e04012ba08a640828b73c52803bf214b])
I20250112 02:52:32.300431 919 raft_consensus.cc:2463] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:32.301008 1706 tablet_loader.cc:96] loaded metadata for tablet 05000c6a48e0450d8e6685ca864f67f0 (table default.Table_692d9e1bfcfd435e8c420696c1e5aae5 [id=1ec97dc2b50a4a3dbbc449559938cead])
I20250112 02:52:32.300909 989 leader_election.cc:304] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:32.301350 1875 raft_consensus.cc:2798] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:32.301491 1875 raft_consensus.cc:491] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:32.301599 1875 raft_consensus.cc:3054] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.305100 1181 tablet_bootstrap.cc:492] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:32.304976 1875 raft_consensus.cc:513] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.305292 1181 ts_tablet_manager.cc:1397] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.001s
I20250112 02:52:32.306049 1609 consensus_queue.cc:1035] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.034s
I20250112 02:52:32.306217 1181 raft_consensus.cc:357] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.306488 1181 raft_consensus.cc:383] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.306595 1181 raft_consensus.cc:738] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.306877 1181 consensus_queue.cc:260] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.307024 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b599e983ec04f53a440d1ab828d12ab" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.307197 776 raft_consensus.cc:2388] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1e775296eb746db8b318f17d26820e6 in current term 1: Already voted for candidate fe36951a83fb4e57b26e187edbcbfe3d in this term.
W20250112 02:52:32.307561 855 leader_election.cc:343] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.307708 855 leader_election.cc:304] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:32.307909 1875 leader_election.cc:290] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.308104 1859 raft_consensus.cc:2743] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.308694 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d0f67141dc46b6a87ca0ad080d5bfe" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:32.308910 919 raft_consensus.cc:3054] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.312732 919 raft_consensus.cc:2463] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:32.313448 989 leader_election.cc:304] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:32.313967 1875 raft_consensus.cc:2798] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:32.314090 1875 raft_consensus.cc:695] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:32.314112 1181 ts_tablet_manager.cc:1428] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250112 02:52:32.314317 1875 consensus_queue.cc:237] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.314994 1181 tablet_bootstrap.cc:492] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:32.315327 1877 leader_election.cc:290] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.316417 1177 tablet_bootstrap.cc:492] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:32.316478 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c530aa0aa64e47ae27d61f341d2bd9" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.316731 775 raft_consensus.cc:2463] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:32.317842 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d0f67141dc46b6a87ca0ad080d5bfe" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.318552 991 leader_election.cc:304] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:32.318990 991 leader_election.cc:304] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
W20250112 02:52:32.319357 991 leader_election.cc:343] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.319653 1866 raft_consensus.cc:3054] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.321435 1181 tablet_bootstrap.cc:654] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.322480 1866 raft_consensus.cc:2743] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.322647 1866 raft_consensus.cc:2798] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:32.322729 1866 raft_consensus.cc:491] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:32.322811 1866 raft_consensus.cc:3054] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.323788 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d0f67141dc46b6a87ca0ad080d5bfe" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
W20250112 02:52:32.330641 991 leader_election.cc:343] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.331780 1859 consensus_queue.cc:1035] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.337095 1177 tablet_bootstrap.cc:654] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.340929 1706 tablet_loader.cc:96] loaded metadata for tablet 058a9ca089cd4967887b0a1127c7d03f (table default.Table_a56064e2c6b449b7b1e32f2a69f2f7fe [id=448877a19b0d4ed2abbc3a796e40eaeb])
W20250112 02:52:32.341459 989 consensus_peers.cc:487] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb0aba0139cf479b80b32f3dfc936535. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:32.341936 1706 tablet_loader.cc:96] loaded metadata for tablet 09d06e2e8c13464db8cf801a292bdc99 (table default.Table_0bdc5f49d4ff4d6fb1290bc2dc0f70d9 [id=e3bb489c53454222a79702b798c0c2f1])
I20250112 02:52:32.342506 1706 tablet_loader.cc:96] loaded metadata for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0])
I20250112 02:52:32.342800 1706 tablet_loader.cc:96] loaded metadata for tablet 13578542e9e741e8817e0ee63b10f1a1 (table default.Table_63abe3ac694a4ed6bf4aef720b542e73 [id=a7ce19c248fe4ac6a6b436773d6c1b4c])
I20250112 02:52:32.354665 1177 tablet_bootstrap.cc:492] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:32.355041 1177 ts_tablet_manager.cc:1397] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.093s user 0.001s sys 0.003s
I20250112 02:52:32.355612 1866 raft_consensus.cc:513] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:32.355778 1181 tablet_bootstrap.cc:492] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:32.355934 1181 ts_tablet_manager.cc:1397] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.002s
I20250112 02:52:32.355851 1177 raft_consensus.cc:357] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.356046 1177 raft_consensus.cc:383] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.356137 1177 raft_consensus.cc:738] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.356722 1866 leader_election.cc:290] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.357223 1181 raft_consensus.cc:357] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.357267 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c530aa0aa64e47ae27d61f341d2bd9" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:32.357419 1181 raft_consensus.cc:383] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.357476 775 raft_consensus.cc:3054] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.357501 1181 raft_consensus.cc:738] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.357717 1181 consensus_queue.cc:260] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.359213 1181 ts_tablet_manager.cc:1428] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20250112 02:52:32.359925 989 leader_election.cc:343] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: c7c530aa0aa64e47ae27d61f341d2bd9
I20250112 02:52:32.360183 1181 tablet_bootstrap.cc:492] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:32.361918 1875 consensus_queue.cc:1035] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.362426 1181 tablet_bootstrap.cc:654] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.343420 1706 tablet_loader.cc:96] loaded metadata for tablet 164f76edcf8f4bd498dcd2c25fd7477d (table default.Table_5e6ad2a2040d41b381bb2653c6f691ec [id=a2024dc6a87341cb87c45eee7ab024b0])
I20250112 02:52:32.363862 1706 tablet_loader.cc:96] loaded metadata for tablet 1811ce8982fd4003a35e89ea34e32746 (table default.Table_3e58cdf6ca254711b8071a2899d2402f [id=6a788a11657f4098bc906e16cd54618e])
I20250112 02:52:32.364442 775 raft_consensus.cc:2463] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:32.364598 1706 tablet_loader.cc:96] loaded metadata for tablet 1a7717cdcb8a4f47b11ba7af54b57e3c (table default.Table_f7fd6e154c104f7ab66a01e7f8202da6 [id=4447b2d49c864e268d7eec91384899cc])
I20250112 02:52:32.365474 991 leader_election.cc:304] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:32.365655 1706 tablet_loader.cc:96] loaded metadata for tablet 1dda52f012794493a102ed1abc718f5b (table default.Table_ab4d7c58924a4b8682b52637c23f125b [id=a622f7a973344a388b3275adbcce6234])
I20250112 02:52:32.365967 1875 raft_consensus.cc:2798] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:32.366107 1875 raft_consensus.cc:695] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:32.366264 1706 tablet_loader.cc:96] loaded metadata for tablet 2052b4536f9745f0a785f330d706c1cf (table default.Table_096703f878244b8dac2dcf187fd23fd6 [id=e1779f88b20942eaa2c640ac24946fff])
I20250112 02:52:32.366427 1875 consensus_queue.cc:237] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:32.366663 1706 tablet_loader.cc:96] loaded metadata for tablet 223eb8b4d1024985844867d5641600d1 (table default.Table_e1d4a3104fc0481287acede4261941c6 [id=231e339ce95d48c99307e09b9de230c9])
W20250112 02:52:32.370669 989 consensus_peers.cc:487] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aadb2ab1d2024f889841ac91e065e064. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:32.356397 1177 consensus_queue.cc:260] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.380636 1177 ts_tablet_manager.cc:1428] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20250112 02:52:32.381186 1177 tablet_bootstrap.cc:492] T ea33e5c1a1fb46599c5836a36b130051 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:32.384016 1177 tablet_bootstrap.cc:654] T ea33e5c1a1fb46599c5836a36b130051 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.387199 1714 consensus_queue.cc:1035] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.390280 1049 raft_consensus.cc:3054] T 7f4e4f1e451b48dc9cdd780300f6888c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.396184 1173 tablet_bootstrap.cc:492] T 2c03b5e612d64314b210c11686fdfa2a P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:32.396365 1173 ts_tablet_manager.cc:1397] T 2c03b5e612d64314b210c11686fdfa2a P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.118s user 0.010s sys 0.018s
I20250112 02:52:32.397082 1173 raft_consensus.cc:357] T 2c03b5e612d64314b210c11686fdfa2a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.397289 1173 raft_consensus.cc:383] T 2c03b5e612d64314b210c11686fdfa2a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.397405 1173 raft_consensus.cc:738] T 2c03b5e612d64314b210c11686fdfa2a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.398068 1173 consensus_queue.cc:260] T 2c03b5e612d64314b210c11686fdfa2a P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.400035 1181 tablet_bootstrap.cc:492] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:32.400323 1181 ts_tablet_manager.cc:1397] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.002s
I20250112 02:52:32.400458 1173 ts_tablet_manager.cc:1428] T 2c03b5e612d64314b210c11686fdfa2a P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250112 02:52:32.400806 1173 tablet_bootstrap.cc:492] T 7ea075826b16475ca474e284fc6bc545 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:32.402362 1706 tablet_loader.cc:96] loaded metadata for tablet 2361bc6bd8164c2daa37be6b0845f4bd (table default.Table_ad5f01f6a5274cc483c8f051059d36c3 [id=8be34ef1e0c54f7fa38ec57fb432b3db])
I20250112 02:52:32.405483 1706 tablet_loader.cc:96] loaded metadata for tablet 23e7a2e88c374e949ced92c557225cbf (table default.Table_cd1f185d82b3474083246ec3702851fc [id=bfa9a504a8d34c86929efb0d15d049d0])
I20250112 02:52:32.404875 1181 raft_consensus.cc:357] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.406152 1181 raft_consensus.cc:383] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.406538 1181 raft_consensus.cc:738] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.407928 1706 tablet_loader.cc:96] loaded metadata for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_849b7f79a5784f5fabed0136deb0a129 [id=386e3565dbf2490997e5cc81d6f02567])
I20250112 02:52:32.407867 1181 consensus_queue.cc:260] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.412885 1866 consensus_queue.cc:1035] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.417210 1866 raft_consensus.cc:491] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.417367 1866 raft_consensus.cc:513] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.417996 1866 leader_election.cc:290] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.418450 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb6d20cfa0453e90e048825114525e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:32.418772 919 raft_consensus.cc:2463] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:32.419416 989 leader_election.cc:304] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:32.419853 1866 raft_consensus.cc:2798] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:32.419956 1866 raft_consensus.cc:491] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:32.420065 1866 raft_consensus.cc:3054] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.425634 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb6d20cfa0453e90e048825114525e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.426321 778 raft_consensus.cc:2463] T 49bb6d20cfa0453e90e048825114525e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:32.427655 1173 tablet_bootstrap.cc:654] T 7ea075826b16475ca474e284fc6bc545 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.430536 1706 tablet_loader.cc:96] loaded metadata for tablet 26577b614e32458e8bc5d6830c8c2e40 (table default.Table_03e3448d809c4fc792d0bff94a713044 [id=5f3cc73b47414c438591107e35a12472])
I20250112 02:52:32.431654 1181 ts_tablet_manager.cc:1428] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
I20250112 02:52:32.433205 1866 raft_consensus.cc:513] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.434340 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb6d20cfa0453e90e048825114525e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:32.434548 919 raft_consensus.cc:3054] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.435170 1866 leader_election.cc:290] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.435388 1181 tablet_bootstrap.cc:492] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:32.436868 1727 consensus_queue.cc:1035] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.437482 919 raft_consensus.cc:2463] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:32.439827 989 leader_election.cc:304] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:32.440135 1866 raft_consensus.cc:2798] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:32.439982 1706 tablet_loader.cc:96] loaded metadata for tablet 27ab58e45d0d44b283019240f44b66bf (table default.Table_d588e6b08cae48e590a4a57002df6323 [id=7a7301e1e2d74a6d9340ee996656f264])
I20250112 02:52:32.440472 1866 raft_consensus.cc:695] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:32.441202 1866 consensus_queue.cc:237] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.445302 1173 tablet_bootstrap.cc:492] T 7ea075826b16475ca474e284fc6bc545 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:32.446758 1706 tablet_loader.cc:96] loaded metadata for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4])
I20250112 02:52:32.449296 1602 consensus_queue.cc:1035] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.449815 1173 ts_tablet_manager.cc:1397] T 7ea075826b16475ca474e284fc6bc545 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.049s user 0.001s sys 0.003s
I20250112 02:52:32.451692 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb6d20cfa0453e90e048825114525e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:32.452020 779 raft_consensus.cc:3054] T 49bb6d20cfa0453e90e048825114525e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.455585 779 raft_consensus.cc:2463] T 49bb6d20cfa0453e90e048825114525e P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:32.455494 1181 tablet_bootstrap.cc:654] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.457417 1706 tablet_loader.cc:96] loaded metadata for tablet 2c03b5e612d64314b210c11686fdfa2a (table default.Table_4934f6022f2441eda88751236a5798be [id=5c4769bfcefd4fe0a0f3c2e315d50c32])
I20250112 02:52:32.459758 1706 tablet_loader.cc:96] loaded metadata for tablet 2d3d291ca3494f05ac23c7fe293e0bd9 (table default.Table_c11c44af47604f9589b15d5c6a97c255 [id=e80fb97e841a4e68a6ab992d897816c1])
I20250112 02:52:32.466675 1852 consensus_queue.cc:1035] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.475098 1181 tablet_bootstrap.cc:492] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:32.475648 1181 ts_tablet_manager.cc:1397] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.000s
I20250112 02:52:32.478029 1877 consensus_queue.cc:1035] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.478344 1173 raft_consensus.cc:357] T 7ea075826b16475ca474e284fc6bc545 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.478592 1173 raft_consensus.cc:383] T 7ea075826b16475ca474e284fc6bc545 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.478690 1173 raft_consensus.cc:738] T 7ea075826b16475ca474e284fc6bc545 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.478960 1173 consensus_queue.cc:260] T 7ea075826b16475ca474e284fc6bc545 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.479447 1173 ts_tablet_manager.cc:1428] T 7ea075826b16475ca474e284fc6bc545 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250112 02:52:32.479959 1852 consensus_queue.cc:1035] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.481242 1173 tablet_bootstrap.cc:492] T 8a12b225131b464494a9f02133d37aea P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:32.481659 1917 log.cc:926] Time spent T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20250112 02:52:32.502876 1706 tablet_loader.cc:96] loaded metadata for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf])
I20250112 02:52:32.504040 1706 tablet_loader.cc:96] loaded metadata for tablet 3b9280db29e54d41af1bed2fa5aa5535 (table default.Table_6671f628f89b428a83eca77e9ca26516 [id=e0577e1a40064437bb7eb4c66fbd914e])
I20250112 02:52:32.505613 1706 tablet_loader.cc:96] loaded metadata for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f])
I20250112 02:52:32.511932 1859 consensus_queue.cc:1035] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.498378 1181 raft_consensus.cc:357] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.521708 1181 raft_consensus.cc:383] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.521837 1181 raft_consensus.cc:738] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.522122 1181 consensus_queue.cc:260] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.531805 1181 ts_tablet_manager.cc:1428] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.056s user 0.003s sys 0.000s
I20250112 02:52:32.532418 1181 tablet_bootstrap.cc:492] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:32.535176 1706 tablet_loader.cc:96] loaded metadata for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24])
I20250112 02:52:32.536463 1706 tablet_loader.cc:96] loaded metadata for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_27f1bff714d947d5b3f42be6bfbe8be2 [id=78dff38cafdc48ddbc46b91a548c2022])
I20250112 02:52:32.537498 1177 tablet_bootstrap.cc:492] T ea33e5c1a1fb46599c5836a36b130051 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:32.537678 1177 ts_tablet_manager.cc:1397] T ea33e5c1a1fb46599c5836a36b130051 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.157s user 0.012s sys 0.020s
I20250112 02:52:32.537705 1706 tablet_loader.cc:96] loaded metadata for tablet 3f5b418c17084420b867498fda79fb0a (table default.Table_b0af74e9ea8a4c5faea64696a16d5a9a [id=ad0be37e6b6f4b9597dcb4cbd50d5b12])
I20250112 02:52:32.538494 1177 raft_consensus.cc:357] T ea33e5c1a1fb46599c5836a36b130051 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.538702 1177 raft_consensus.cc:383] T ea33e5c1a1fb46599c5836a36b130051 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.538964 1177 raft_consensus.cc:738] T ea33e5c1a1fb46599c5836a36b130051 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.539242 1177 consensus_queue.cc:260] T ea33e5c1a1fb46599c5836a36b130051 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.539618 1181 tablet_bootstrap.cc:654] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.541172 1706 tablet_loader.cc:96] loaded metadata for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_2b5aeec3b0bf40aea08010f65e422c2c [id=e292141365e9453fa1816e0f313b2300])
I20250112 02:52:32.545830 1173 tablet_bootstrap.cc:654] T 8a12b225131b464494a9f02133d37aea P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.547912 1734 consensus_queue.cc:1035] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.552114 1706 tablet_loader.cc:96] loaded metadata for tablet 42444e99aba84a7b873bc8cdf954a6dc (table default.Table_c7107062f9574066bf9a4a362f9b3cfb [id=70e9b6c9e20f40eab186f042f21ad2ec])
I20250112 02:52:32.561103 1706 tablet_loader.cc:96] loaded metadata for tablet 46d1beac4410422bbf1e912d35999f3d (table default.Table_ed3c26ab0711404581dcd62adb343dde [id=3a0f5ac3329d4b60befce41c91f5ae5c])
I20250112 02:52:32.561930 1706 tablet_loader.cc:96] loaded metadata for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264])
I20250112 02:52:32.562501 1859 consensus_queue.cc:1035] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.563584 1706 tablet_loader.cc:96] loaded metadata for tablet 48634d6d78784fae98512b242e5476df (table default.Table_977c68d65cd443ef936f3dd77b445154 [id=fee2f9e65e314ce9a1fc1fb6dfcde317])
I20250112 02:52:32.565691 1706 tablet_loader.cc:96] loaded metadata for tablet 4916ae3917084cfe8c93240a8857be0e (table default.Table_69f363c4da234792886b544823b3214e [id=c7052afaf6284ef0972835c1f46cb829])
I20250112 02:52:32.568687 1706 tablet_loader.cc:96] loaded metadata for tablet 4a57eb366f594f93a488c8250e195014 (table default.Table_7ee8a748c60d439088423dcb284b17ad [id=f836cdee435949d387336c9a52b6ed1b])
I20250112 02:52:32.571089 1173 tablet_bootstrap.cc:492] T 8a12b225131b464494a9f02133d37aea P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:32.571979 1173 ts_tablet_manager.cc:1397] T 8a12b225131b464494a9f02133d37aea P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.091s user 0.000s sys 0.003s
I20250112 02:52:32.572450 1181 tablet_bootstrap.cc:492] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:32.572665 1181 ts_tablet_manager.cc:1397] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.001s
I20250112 02:52:32.573781 1706 tablet_loader.cc:96] loaded metadata for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c])
I20250112 02:52:32.573725 1181 raft_consensus.cc:357] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.573951 1181 raft_consensus.cc:383] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.574041 1181 raft_consensus.cc:738] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.574282 1181 consensus_queue.cc:260] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.574817 1181 ts_tablet_manager.cc:1428] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:32.575331 1181 tablet_bootstrap.cc:492] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:32.576298 1706 tablet_loader.cc:96] loaded metadata for tablet 4fc8658b0efe43bdb339edbff64973f6 (table default.Table_dc563f08a790498993b01afeec904975 [id=955093b0cd9a49bdbdc0517dc693a41d])
I20250112 02:52:32.577174 1181 tablet_bootstrap.cc:654] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.577354 1706 tablet_loader.cc:96] loaded metadata for tablet 54215c55966a46e1ab96e352c4e3764e (table default.Table_6e77252e817f43399c9b3f5b221795ad [id=1fa1474ab8804667b22e5f30c48c1630])
I20250112 02:52:32.577919 1929 raft_consensus.cc:491] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 436220c23fa948efb5b6ca15875d8b0e)
I20250112 02:52:32.578048 1929 raft_consensus.cc:513] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:32.578105 1706 tablet_loader.cc:96] loaded metadata for tablet 55fb1a197d7e40689ccad82a44492371 (table default.Table_f3695b00a89b4353903587a6c1418d8c [id=d1781738b8a74f139b09863918cbb21d])
I20250112 02:52:32.579156 1706 tablet_loader.cc:96] loaded metadata for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0])
I20250112 02:52:32.579850 1706 tablet_loader.cc:96] loaded metadata for tablet 5a3af0ea4e954053ae89814f3e74e6b7 (table default.Table_525750dad2b7461586d070dbaa74d0b1 [id=7d048aa907a74605964b68726ac5d759])
I20250112 02:52:32.580767 1706 tablet_loader.cc:96] loaded metadata for tablet 5cc414d6420c41839621ab858067548b (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4])
I20250112 02:52:32.581995 1706 tablet_loader.cc:96] loaded metadata for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_0dec5de812ab4ea48f1b7a8f4cb0cc4f [id=2f43ebd749304c0287769224ac6b12c7])
I20250112 02:52:32.584703 1052 raft_consensus.cc:3054] T 039a4734358646508093200d59d30013 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.585002 1181 tablet_bootstrap.cc:492] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:32.584964 1706 tablet_loader.cc:96] loaded metadata for tablet 6117dc8b32a64535a2f4167a03bed0d2 (table default.Table_365739770a11434dbf260ea912e7ea81 [id=868adb627654450c936ef198e49d853b])
I20250112 02:52:32.585126 1181 ts_tablet_manager.cc:1397] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20250112 02:52:32.585789 1181 raft_consensus.cc:357] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.585958 1181 raft_consensus.cc:383] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.586038 1181 raft_consensus.cc:738] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.587373 1181 consensus_queue.cc:260] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.587958 1181 ts_tablet_manager.cc:1428] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:32.589453 1181 tablet_bootstrap.cc:492] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:32.588217 1706 tablet_loader.cc:96] loaded metadata for tablet 6171b8ff2bfe41d598b0cb2535799a11 (table default.Table_e14b6c6d907942ebb7f8410e18d38ce2 [id=3097d5437937423e95b710c00473b6fa])
I20250112 02:52:32.591113 1706 tablet_loader.cc:96] loaded metadata for tablet 61eac6b572ff47c185768bee7490edf7 (table default.Table_e8e62bd2fffe4f50849f722dbece2a2c [id=3603f999c77a48dca737bc8ff74eb9de])
W20250112 02:52:32.592016 439 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:38771, user_credentials={real_user=slave}} blocked reactor thread for 55306.9us
I20250112 02:52:32.592695 1177 ts_tablet_manager.cc:1428] T ea33e5c1a1fb46599c5836a36b130051 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.055s user 0.002s sys 0.000s
I20250112 02:52:32.593255 1181 tablet_bootstrap.cc:654] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.592574 1706 tablet_loader.cc:96] loaded metadata for tablet 62f4bd2df0ce45dabc55373f1154d3ac (table default.Table_7921b99e24f84cceb58128f18925ad49 [id=0525f87eccbb4a1d8726d18a00f9a4e4])
I20250112 02:52:32.603260 1177 tablet_bootstrap.cc:492] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:32.604411 1929 leader_election.cc:290] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 436220c23fa948efb5b6ca15875d8b0e (127.0.104.62:42341), f4a49a5756b143fcb767c3f41463f8d9 (127.0.104.61:38771)
I20250112 02:52:32.606082 1177 tablet_bootstrap.cc:654] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.603825 1173 raft_consensus.cc:357] T 8a12b225131b464494a9f02133d37aea P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.607091 1173 raft_consensus.cc:383] T 8a12b225131b464494a9f02133d37aea P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.608580 1173 raft_consensus.cc:738] T 8a12b225131b464494a9f02133d37aea P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.609210 1173 consensus_queue.cc:260] T 8a12b225131b464494a9f02133d37aea P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.610082 1706 tablet_loader.cc:96] loaded metadata for tablet 709517ae520640208abe85fa14a5301e (table default.Table_b8af9fadb7b64c07afb49c07cf3ca5f9 [id=c7f00d3d40fb4a469c57e3f82983ff53])
I20250112 02:52:32.615885 1181 tablet_bootstrap.cc:492] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:32.616020 1181 ts_tablet_manager.cc:1397] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.001s
I20250112 02:52:32.616673 1181 raft_consensus.cc:357] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.616868 1181 raft_consensus.cc:383] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.616936 1181 raft_consensus.cc:738] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.617141 1181 consensus_queue.cc:260] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.617619 1181 ts_tablet_manager.cc:1428] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 02:52:32.617893 1181 tablet_bootstrap.cc:492] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:32.619165 1173 ts_tablet_manager.cc:1428] T 8a12b225131b464494a9f02133d37aea P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.047s user 0.002s sys 0.000s
I20250112 02:52:32.619165 1859 consensus_queue.cc:1035] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.620349 1173 tablet_bootstrap.cc:492] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:32.608752 1734 consensus_queue.cc:1035] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.625629 1181 tablet_bootstrap.cc:654] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.630661 1718 consensus_queue.cc:1035] T 039a4734358646508093200d59d30013 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.635167 1173 tablet_bootstrap.cc:654] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.645704 1706 tablet_loader.cc:96] loaded metadata for tablet 7320ac55dbfa47fe851fff94d32e1ffe (table default.Table_f8a40c27bebc483cbf62d5d0a4c7fb24 [id=8cc378b0da6e4b77bc2c6822cd0844fe])
I20250112 02:52:32.646734 1706 tablet_loader.cc:96] loaded metadata for tablet 763f77385e404c149c16edabe469c299 (table default.Table_6ff2131cc2144af7b6c223f14e9eb25a [id=fc8a29e5f2e24a4784009e85a98a40bd])
I20250112 02:52:32.647701 1706 tablet_loader.cc:96] loaded metadata for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13])
I20250112 02:52:32.648433 1706 tablet_loader.cc:96] loaded metadata for tablet 781fe3514f4c4f80b1ee49278502ccc1 (table default.Table_76e4d0c54155412296bc900e03cd40bd [id=aecf277dcd794bf3bf27d58c9b96dd12])
I20250112 02:52:32.649123 1706 tablet_loader.cc:96] loaded metadata for tablet 795ac7eaf81d48bb926865d9ebb388b6 (table default.Table_178713b464bb4f16970851646a710574 [id=d3f31247f0e848f8b2dfa18f3437ca63])
I20250112 02:52:32.649824 1706 tablet_loader.cc:96] loaded metadata for tablet 79c6bdd89f5a44efb56d65ccb724308b (table default.Table_ad69cfacaf7b4050ab2d3d6ed237e7d0 [id=9e72fad141db43c09c7e4692769b6eba])
W20250112 02:52:32.649889 1345 leader_election.cc:336] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f4a49a5756b143fcb767c3f41463f8d9 (127.0.104.61:38771): Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection refused (error 111)
I20250112 02:52:32.653059 1173 tablet_bootstrap.cc:492] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:32.653299 1173 ts_tablet_manager.cc:1397] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20250112 02:52:32.654585 1706 tablet_loader.cc:96] loaded metadata for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_5adbd32e9a0c4736844af4c72b7787fd [id=3a629cb4a4104e30b0beb8f729ba0a22])
I20250112 02:52:32.655006 1173 raft_consensus.cc:357] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.656518 1173 raft_consensus.cc:383] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.656622 1173 raft_consensus.cc:738] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.656971 1173 consensus_queue.cc:260] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.660229 1173 ts_tablet_manager.cc:1428] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250112 02:52:32.663081 1866 raft_consensus.cc:491] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.663221 1866 raft_consensus.cc:513] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.663815 1866 leader_election.cc:290] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.664961 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea33e5c1a1fb46599c5836a36b130051" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:32.665170 918 raft_consensus.cc:2463] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:32.665685 989 leader_election.cc:304] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:32.666431 1866 raft_consensus.cc:2798] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:32.666527 1866 raft_consensus.cc:491] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:32.666600 1866 raft_consensus.cc:3054] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.669265 1866 raft_consensus.cc:513] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.670393 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea33e5c1a1fb46599c5836a36b130051" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:32.670586 918 raft_consensus.cc:3054] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.671348 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea33e5c1a1fb46599c5836a36b130051" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.671813 775 raft_consensus.cc:2463] T ea33e5c1a1fb46599c5836a36b130051 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:32.673090 918 raft_consensus.cc:2463] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:32.676329 989 leader_election.cc:304] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:32.676582 1877 raft_consensus.cc:2798] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:32.676712 1173 tablet_bootstrap.cc:492] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:32.676748 1877 raft_consensus.cc:695] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:32.676990 1877 consensus_queue.cc:237] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.677139 1859 raft_consensus.cc:491] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.677268 1859 raft_consensus.cc:513] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:32.678057 1859 leader_election.cc:290] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:32.678931 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc414d6420c41839621ab858067548b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:32.679514 854 leader_election.cc:343] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.679548 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea33e5c1a1fb46599c5836a36b130051" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:32.679740 776 raft_consensus.cc:3054] T ea33e5c1a1fb46599c5836a36b130051 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.679807 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc414d6420c41839621ab858067548b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:32.680262 855 leader_election.cc:343] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.680459 855 leader_election.cc:304] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:32.680919 1859 raft_consensus.cc:2743] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.682358 776 raft_consensus.cc:2463] T ea33e5c1a1fb46599c5836a36b130051 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:32.683287 1173 tablet_bootstrap.cc:654] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.684672 1706 tablet_loader.cc:96] loaded metadata for tablet 7ad9c47fcb534619949971796b60eba2 (table default.Table_569c0e413a014e52bcb98a9f4f644479 [id=75c94dcaccf344abbac5808bab28a673])
I20250112 02:52:32.685755 1706 tablet_loader.cc:96] loaded metadata for tablet 7b9c845571f44c8eb8753090d63df45f (table default.Table_7077f9a718664d8b915ee4003502306a [id=6df53b1d0c7a4fa1a18b3202741630b5])
I20250112 02:52:32.686672 1866 leader_election.cc:290] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.688586 1706 tablet_loader.cc:96] loaded metadata for tablet 7cd9985cd20148f195a857c478b0755f (table default.Table_9b65dd291c9f4fcdaa6d5e37f04d24ae [id=e63daf724d874c89badafb8d59653f74])
I20250112 02:52:32.689078 1173 tablet_bootstrap.cc:492] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:32.689210 1173 ts_tablet_manager.cc:1397] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.004s
I20250112 02:52:32.689328 1706 tablet_loader.cc:96] loaded metadata for tablet 7e1954e7859f422f9690fe4a9384a8bb (table default.Table_829049fe22b74e62a87dd1bed65fb3fe [id=2e892eba68234ffca0be4ef3e49cf309])
I20250112 02:52:32.689993 1706 tablet_loader.cc:96] loaded metadata for tablet 7ea075826b16475ca474e284fc6bc545 (table default.Table_009a6976f6db4f218e3c3373368a58d7 [id=c5005669bb5f4c029d673bd39cc1535d])
I20250112 02:52:32.694518 1181 tablet_bootstrap.cc:492] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:32.694799 1181 ts_tablet_manager.cc:1397] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.077s user 0.007s sys 0.009s
I20250112 02:52:32.694949 1177 tablet_bootstrap.cc:492] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:32.694890 1173 raft_consensus.cc:357] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.695047 1173 raft_consensus.cc:383] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.695150 1173 raft_consensus.cc:738] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.695166 1177 ts_tablet_manager.cc:1397] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.092s user 0.002s sys 0.012s
I20250112 02:52:32.695822 1177 raft_consensus.cc:357] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.696048 1177 raft_consensus.cc:383] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.696156 1177 raft_consensus.cc:738] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.696478 1177 consensus_queue.cc:260] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.696604 1181 raft_consensus.cc:357] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:32.696888 1181 raft_consensus.cc:383] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.697037 1181 raft_consensus.cc:738] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.697397 1181 consensus_queue.cc:260] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:32.698508 1173 consensus_queue.cc:260] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.698964 1181 ts_tablet_manager.cc:1428] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250112 02:52:32.699151 1173 ts_tablet_manager.cc:1428] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20250112 02:52:32.700786 1173 tablet_bootstrap.cc:492] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:32.700757 1859 raft_consensus.cc:491] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.700877 1859 raft_consensus.cc:513] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.701503 1859 leader_election.cc:290] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.702082 1181 tablet_bootstrap.cc:492] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:32.702025 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7cd0927c6a4401cbe69f55d4db9963a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.702049 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7cd0927c6a4401cbe69f55d4db9963a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:32.702419 855 leader_election.cc:343] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.702570 1177 ts_tablet_manager.cc:1428] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
W20250112 02:52:32.702584 854 leader_election.cc:343] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.702683 854 leader_election.cc:304] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:32.703550 1859 raft_consensus.cc:2743] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.709858 1177 tablet_bootstrap.cc:492] T 4a57eb366f594f93a488c8250e195014 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:32.710301 1706 tablet_loader.cc:96] loaded metadata for tablet 7f3be6bcbe8a45f0a8076b4842d90dc2 (table default.Table_6d12502c6e714726b9d807690fd21ea4 [id=9b5607155d264cf2879d18305398df11])
I20250112 02:52:32.711048 1706 tablet_loader.cc:96] loaded metadata for tablet 82b351c563d749df806ab5d0331a4ef2 (table default.Table_4603068c01544e2e965dca5177e36b40 [id=8502f21e9b61436093db1241cfbb5ec9])
I20250112 02:52:32.710986 1181 tablet_bootstrap.cc:654] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.711299 1173 tablet_bootstrap.cc:654] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.712002 1706 tablet_loader.cc:96] loaded metadata for tablet 8429a3d8636d4e2ca4c1191f85b9266b (table default.Table_328664e3c01f483ca8e5ff50a497f68e [id=1f5c5f9f5e664250898a3935787b6b51])
I20250112 02:52:32.712417 1706 tablet_loader.cc:96] loaded metadata for tablet 856fa83dedfd4b049e3e0d91d7ac5e40 (table default.Table_4657847b84384e628194a8ff43e183a1 [id=85db347bdf4a4ff583062f3dddd20d8a])
I20250112 02:52:32.714926 1706 tablet_loader.cc:96] loaded metadata for tablet 88d69632845547f588662dd56c8c194e (table default.Table_d3adef5e4ff541dabd0661d1c0997a71 [id=98b0d39d324c4b2b8e83dbb31776a5cd])
I20250112 02:52:32.715291 1706 tablet_loader.cc:96] loaded metadata for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_feaebd2f4cbb47b6b24ff79614994d6f [id=63c20c66463d4a30b7e1558fecce2704])
I20250112 02:52:32.715710 1706 tablet_loader.cc:96] loaded metadata for tablet 89b70dce4d1d4e29997a18297ac0f82a (table default.Table_f9db9e236376446f8210cc4c9708ec6a [id=2fa03135646d4939b6efc7811ee4df10])
I20250112 02:52:32.716910 1177 tablet_bootstrap.cc:654] T 4a57eb366f594f93a488c8250e195014 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.717517 1706 tablet_loader.cc:96] loaded metadata for tablet 8a88463991294bd8a335337f8b072c3d (table default.Table_56ea07abda424090a7942efabc871092 [id=44f7673679d14145a8cf65abd82094ce])
I20250112 02:52:32.718364 1706 tablet_loader.cc:96] loaded metadata for tablet 8d3a4e76b724452c8b238aae0198e297 (table default.Table_add1776e850b4a1bbd63f05707b51873 [id=ec37a56b10ba407b86e851f07bddf528])
I20250112 02:52:32.720381 1718 consensus_queue.cc:1035] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.720870 1706 tablet_loader.cc:96] loaded metadata for tablet 8ee98f055024429eafaa29bb961cab87 (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1])
I20250112 02:52:32.725172 1054 raft_consensus.cc:3054] T 8a12b225131b464494a9f02133d37aea P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.728418 1859 consensus_queue.cc:1035] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.736402 1706 tablet_loader.cc:96] loaded metadata for tablet 9af55dab202444358ea1a0674d37946d (table default.Table_26dfe346379948b39f90b31653c2a463 [id=6c6f30e60e79451397841d090d11d8b7])
I20250112 02:52:32.736819 1706 tablet_loader.cc:96] loaded metadata for tablet 9b81aee21510452881948e41d7aba9be (table default.Table_29a6037a21724b5f9f0e0ee6a598a3dd [id=8b6988397e544a37b869f1e782095b64])
I20250112 02:52:32.740330 1706 tablet_loader.cc:96] loaded metadata for tablet 9bbef42e11a843b18ed582ab4d04a92a (table default.Table_c816af31a1e945518fd886cbf8b8b885 [id=93ab7e5945514c0aa85b411f15483f3d])
I20250112 02:52:32.741940 1706 tablet_loader.cc:96] loaded metadata for tablet 9ded8bcdda914ea6bc4f9bb92d413dee (table default.Table_b0d877b624604ddc93c56bec6e05261f [id=510ea5b8b1ae461faaa7c9787162f29a])
I20250112 02:52:32.744493 1706 tablet_loader.cc:96] loaded metadata for tablet 9f0314aa03b1473ebbe05ce73adf7838 (table default.Table_9f27fd5786f84a8084cdf01f95f4519b [id=0e7297e807b545fa8482864ab59eeb41])
I20250112 02:52:32.752988 1706 tablet_loader.cc:96] loaded metadata for tablet a3ffdc237f8640f0b378809f93fc6292 (table default.Table_fc264b6a06584c7ea757f5d18e05e2bc [id=b560a3488e3d4db58abc7d81d6094d86])
I20250112 02:52:32.754020 1706 tablet_loader.cc:96] loaded metadata for tablet a437952889494b0e820e41fdfffe708d (table default.Table_150748c55de444749d147c320c6d8c0c [id=39ac22eea1dd45de96d7e1829ad9844d])
I20250112 02:52:32.755627 1706 tablet_loader.cc:96] loaded metadata for tablet ab2e31a65294407bb62b13e92898a7d2 (table default.Table_3321055e22c3410eb5e5fe8be62e004c [id=f8cd4289f89a4a748c26a7a01236406e])
I20250112 02:52:32.758738 1714 consensus_queue.cc:1035] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.760815 1706 tablet_loader.cc:96] loaded metadata for tablet abfaad67d5e34c0a8338dccc0ef05466 (table default.Table_045e5b104dc64217aa39c22534007cab [id=2c0e94e8c5774da1b8130c7651d94500])
I20250112 02:52:32.761631 1173 tablet_bootstrap.cc:492] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:32.761813 1173 ts_tablet_manager.cc:1397] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.061s user 0.000s sys 0.010s
I20250112 02:52:32.762526 1173 raft_consensus.cc:357] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.762789 1173 raft_consensus.cc:383] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.762640 1052 raft_consensus.cc:3054] T 2c03b5e612d64314b210c11686fdfa2a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.762885 1173 raft_consensus.cc:738] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.762955 1734 consensus_queue.cc:1035] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.763134 1173 consensus_queue.cc:260] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.763854 1181 tablet_bootstrap.cc:492] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:32.763972 1181 ts_tablet_manager.cc:1397] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.000s
I20250112 02:52:32.764606 1181 raft_consensus.cc:357] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.764822 1181 raft_consensus.cc:383] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.764884 1181 raft_consensus.cc:738] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.765101 1181 consensus_queue.cc:260] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.765573 1181 ts_tablet_manager.cc:1428] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:32.765787 1173 ts_tablet_manager.cc:1428] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250112 02:52:32.765870 1181 tablet_bootstrap.cc:492] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:32.766057 1173 tablet_bootstrap.cc:492] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:32.767709 1173 tablet_bootstrap.cc:654] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.768046 1734 consensus_queue.cc:1035] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.769534 1181 tablet_bootstrap.cc:654] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.775804 1706 tablet_loader.cc:96] loaded metadata for tablet afd6d31155034908ac44a90c3a2ea7d6 (table default.Table_24cecfd421cd4047972cc0a6255ef0c0 [id=5cec72cf833347f3bfbca061f8feca41])
I20250112 02:52:32.777004 1706 tablet_loader.cc:96] loaded metadata for tablet b124c15736084e6d8f69b4fb7c41cf39 (table default.Table_f0fb142b9aab490bb067d3cf64dc47b1 [id=1991c0821ef4478eb0eb9282ab772c0c])
I20250112 02:52:32.777823 1706 tablet_loader.cc:96] loaded metadata for tablet b2ccb88108f24e36ba4b7ebf36398950 (table default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6])
I20250112 02:52:32.778471 1706 tablet_loader.cc:96] loaded metadata for tablet b3c5a861a90840d0a33b780ec75bffd2 (table default.Table_c26869f6cbd540fe980dcf207b5c729d [id=cdd6a2097792484d84bca856f30d7ac1])
I20250112 02:52:32.779662 1706 tablet_loader.cc:96] loaded metadata for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190])
I20250112 02:52:32.781406 1857 raft_consensus.cc:491] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.781571 1857 raft_consensus.cc:513] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.782151 1857 leader_election.cc:290] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.782572 1877 raft_consensus.cc:491] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.782618 1706 tablet_loader.cc:96] loaded metadata for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062])
I20250112 02:52:32.782713 1877 raft_consensus.cc:513] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.782768 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6117dc8b32a64535a2f4167a03bed0d2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.782963 1706 tablet_loader.cc:96] loaded metadata for tablet b8daf4ef3593410eae87672402292be6 (table default.Table_c0aaadb5e69e46e8983f4d289a44afe2 [id=08299ab225a94e37a67722107a5ff4ce])
I20250112 02:52:32.783248 1877 leader_election.cc:290] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
W20250112 02:52:32.783324 855 leader_election.cc:343] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.784298 1706 tablet_loader.cc:96] loaded metadata for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_9487626cbda14256af1f17d2675b5d33 [id=485dd281891f45e8b407b306572d6e29])
I20250112 02:52:32.784368 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7243c6329d4f11aa25eea2b7b67683" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.784550 775 raft_consensus.cc:2463] T ed7243c6329d4f11aa25eea2b7b67683 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:32.784950 991 leader_election.cc:304] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:32.785599 1877 raft_consensus.cc:2798] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:32.785696 1877 raft_consensus.cc:491] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:32.785737 1706 tablet_loader.cc:96] loaded metadata for tablet b9d525dae23744a4adfc2f58cb9945ce (table default.Table_6f3879f2469d4655a6d4436362d0c3bb [id=bb0ba72d0ee747a3945283096d717008])
I20250112 02:52:32.785765 1877 raft_consensus.cc:3054] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.786125 1706 tablet_loader.cc:96] loaded metadata for tablet bab4978edea0445fb0456e88e7fb4c44 (table default.Table_9dd5993a01b44a0abe33a5738c8d5c2b [id=6a7dcf7429b24699b036c53b3005051f])
I20250112 02:52:32.787009 1706 tablet_loader.cc:96] loaded metadata for tablet bb4ef0220dde43569452379639759db5 (table default.Table_f7421d0779924239a3b493954ef748b7 [id=b42fa8fc65e344f989a98ae411e916ed])
I20250112 02:52:32.787405 1706 tablet_loader.cc:96] loaded metadata for tablet bd5b3c712c0f448ea9b14ab93dd731ab (table default.Table_9c10574f7c7d435890eb9a24af23bcb9 [id=c0e16b77525146a28b8182137bc81566])
I20250112 02:52:32.788097 1173 tablet_bootstrap.cc:492] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:32.788223 1173 ts_tablet_manager.cc:1397] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.004s
I20250112 02:52:32.788859 1877 raft_consensus.cc:513] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.788924 1173 raft_consensus.cc:357] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.789105 1173 raft_consensus.cc:383] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.789322 1173 raft_consensus.cc:738] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.789563 1173 consensus_queue.cc:260] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.790073 1181 tablet_bootstrap.cc:492] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:32.790002 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7243c6329d4f11aa25eea2b7b67683" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:32.790238 919 raft_consensus.cc:3054] T ed7243c6329d4f11aa25eea2b7b67683 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.790285 1173 ts_tablet_manager.cc:1428] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 02:52:32.790990 1173 tablet_bootstrap.cc:492] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:32.792554 1173 tablet_bootstrap.cc:654] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.792824 919 raft_consensus.cc:2463] T ed7243c6329d4f11aa25eea2b7b67683 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:32.793352 989 leader_election.cc:304] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:32.793653 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7243c6329d4f11aa25eea2b7b67683" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:32.793887 775 raft_consensus.cc:3054] T ed7243c6329d4f11aa25eea2b7b67683 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.794087 1866 raft_consensus.cc:2798] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:32.794237 1866 raft_consensus.cc:695] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:32.794459 1866 consensus_queue.cc:237] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.796833 775 raft_consensus.cc:2463] T ed7243c6329d4f11aa25eea2b7b67683 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:32.798277 1181 ts_tablet_manager.cc:1397] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.008s
I20250112 02:52:32.801884 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7243c6329d4f11aa25eea2b7b67683" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:32.802085 918 raft_consensus.cc:2371] T ed7243c6329d4f11aa25eea2b7b67683 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1. Re-sending same reply.
I20250112 02:52:32.802953 1877 leader_election.cc:290] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.802450 1181 raft_consensus.cc:357] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.803375 1181 raft_consensus.cc:383] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.803814 1706 tablet_loader.cc:96] loaded metadata for tablet bf58fb4538d946e1818a5d31d2a988e2 (table default.Table_6d358fc697c74529822f0b2b70321a02 [id=9b36703d61a046799cb709c7618b1fbd])
I20250112 02:52:32.804011 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6117dc8b32a64535a2f4167a03bed0d2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:32.803920 1181 raft_consensus.cc:738] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.804497 1706 tablet_loader.cc:96] loaded metadata for tablet c04b64bc2f8348a086ed3c4e59f09b16 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840])
W20250112 02:52:32.804648 854 leader_election.cc:343] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.804788 854 leader_election.cc:304] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:32.805264 1857 raft_consensus.cc:2743] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.805624 1706 tablet_loader.cc:96] loaded metadata for tablet c29a9ca43e034b43a00207e5847bbee3 (table default.Table_9284616ca0cd499cb51df0322db9948f [id=7f36d8ba6d864d32ad84be06ca6682ee])
I20250112 02:52:32.805730 1181 consensus_queue.cc:260] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.809382 1706 tablet_loader.cc:96] loaded metadata for tablet c69587c5a09d40f0baba5c2f6e8a7b41 (table default.Table_1216acc1fa6b4e51a55a89ffe22bcc9a [id=b5457e4343ee43618c8309e737b69ec3])
I20250112 02:52:32.810943 1706 tablet_loader.cc:96] loaded metadata for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a])
I20250112 02:52:32.811444 1181 ts_tablet_manager.cc:1428] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.012s user 0.002s sys 0.002s
I20250112 02:52:32.812572 1706 tablet_loader.cc:96] loaded metadata for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_613140584042445b9d71a3d05cee62c1 [id=f95ac050a4b840a19a8175caade87d48])
I20250112 02:52:32.813400 1173 tablet_bootstrap.cc:492] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:32.813491 1181 tablet_bootstrap.cc:492] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:32.813611 1173 ts_tablet_manager.cc:1397] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.001s
I20250112 02:52:32.814357 1173 raft_consensus.cc:357] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:32.814615 1173 raft_consensus.cc:383] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.815634 1718 consensus_queue.cc:1035] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:32.814723 1173 raft_consensus.cc:738] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.817126 1173 consensus_queue.cc:260] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:32.818581 1173 ts_tablet_manager.cc:1428] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 02:52:32.819002 1173 tablet_bootstrap.cc:492] T a85300a9aa1c4c50a3957a004bea7933 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:32.821557 1173 tablet_bootstrap.cc:654] T a85300a9aa1c4c50a3957a004bea7933 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.825879 1706 tablet_loader.cc:96] loaded metadata for tablet cec5346e13164274ae7bc3b6aa241f61 (table default.Table_4965c2235c4e422889169067ff632d29 [id=8dd987fcd63c4ee7a5fcabd32affaf3d])
I20250112 02:52:32.826310 1706 tablet_loader.cc:96] loaded metadata for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55])
I20250112 02:52:32.828583 1706 tablet_loader.cc:96] loaded metadata for tablet d63d411f906b4d4eb68343bdcc493e05 (table default.Table_e80769df10444dd5ae4bc23a8c6cc09e [id=d91a9f9e1f76482e99d4f10c0568f2e3])
I20250112 02:52:32.829313 1181 tablet_bootstrap.cc:654] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.830118 1706 tablet_loader.cc:96] loaded metadata for tablet d8eb0efc5d7f4c72bdfc7531a444d42a (table default.Table_d561e1740cc545dc8aa4babb76b45c37 [id=de27aeff6b3a49b3956498ea5015ff33])
I20250112 02:52:32.835064 1706 tablet_loader.cc:96] loaded metadata for tablet e2f1f1a76e5f4308bf56dd647f02c6d8 (table default.Table_672d7452ad6d4c99ae9f2ae77b592f78 [id=33dad7ab4106404aaf33266ed1ea5635])
I20250112 02:52:32.835650 1706 tablet_loader.cc:96] loaded metadata for tablet e3ee5d8a17874d7ba54de51568e7d2e6 (table default.Table_83177e7dc14645dbb0e5ef573e80e72f [id=2c077da077834ae4bcc84efdd9e89120])
I20250112 02:52:32.836434 1706 tablet_loader.cc:96] loaded metadata for tablet e477fc157aed4563923d85d66014293a (table default.Table_e495ee84abe3479c80c8ab2e14f86c96 [id=f5eb0a253e444d76978a22250c163a54])
I20250112 02:52:32.837208 1706 tablet_loader.cc:96] loaded metadata for tablet e4b7c2efd7ea402e8d19038f96b9675c (table default.Table_03ccb002507e4b93b1b413ae70602cfc [id=594044c9b27a441cb1002a7b35a6a846])
I20250112 02:52:32.838027 1706 tablet_loader.cc:96] loaded metadata for tablet e4d42f1611de49d6b6e5aaa9be907054 (table default.Table_7926c082d6c04315811ea3b4e1579e8c [id=c01b653cd9fe45b2af721eb97936c5d4])
I20250112 02:52:32.844163 1706 tablet_loader.cc:96] loaded metadata for tablet e933b58dab0b45e8bf1c49eebb1c53e3 (table default.Table_46a03fa2d75e4541a98a1b3d91f85395 [id=e184c10af82240f5b1e29fc226269bf0])
I20250112 02:52:32.845167 1173 tablet_bootstrap.cc:492] T a85300a9aa1c4c50a3957a004bea7933 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:32.845338 1173 ts_tablet_manager.cc:1397] T a85300a9aa1c4c50a3957a004bea7933 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.000s
I20250112 02:52:32.846688 1173 raft_consensus.cc:357] T a85300a9aa1c4c50a3957a004bea7933 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.846935 1173 raft_consensus.cc:383] T a85300a9aa1c4c50a3957a004bea7933 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.847043 1173 raft_consensus.cc:738] T a85300a9aa1c4c50a3957a004bea7933 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.847290 1173 consensus_queue.cc:260] T a85300a9aa1c4c50a3957a004bea7933 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.847836 1173 ts_tablet_manager.cc:1428] T a85300a9aa1c4c50a3957a004bea7933 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 02:52:32.848316 1706 tablet_loader.cc:96] loaded metadata for tablet ea18666ec324403b9f10a099688438b8 (table default.Table_e1b086de31974b33b7ac824ebeb757ee [id=36b5d7e7c4444808921063d46a50da46])
I20250112 02:52:32.849051 1706 tablet_loader.cc:96] loaded metadata for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75])
I20250112 02:52:32.849538 1706 tablet_loader.cc:96] loaded metadata for tablet ecfc7d8ba5e4406ebc61f37e6d500473 (table default.Table_79dbb2fcd6fe4c939bc0637b4ddf1b1e [id=44975f62b4f9429981075254868cb170])
I20250112 02:52:32.849579 1718 raft_consensus.cc:491] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.849763 1718 raft_consensus.cc:513] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.850070 1181 tablet_bootstrap.cc:492] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:32.850410 1181 ts_tablet_manager.cc:1397] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.003s
I20250112 02:52:32.850451 1718 leader_election.cc:290] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:32.850585 1173 tablet_bootstrap.cc:492] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:32.851370 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61593969d2234ce3a6089027b067d922" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:32.851652 919 raft_consensus.cc:2463] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:32.851977 1181 raft_consensus.cc:357] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.852151 713 leader_election.cc:304] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:32.852331 1181 raft_consensus.cc:383] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.852516 1718 raft_consensus.cc:2798] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:32.852465 1181 raft_consensus.cc:738] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.852608 1718 raft_consensus.cc:491] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:32.852699 1718 raft_consensus.cc:3054] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.852947 1173 tablet_bootstrap.cc:654] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.852902 1181 consensus_queue.cc:260] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.853624 1706 tablet_loader.cc:96] loaded metadata for tablet ed7243c6329d4f11aa25eea2b7b67683 (table default.Table_f89a1b732e34437294f2a7cdc052b6ea [id=7d79cefc22aa44309a4fed4d8198c79f])
I20250112 02:52:32.854003 1706 tablet_loader.cc:96] loaded metadata for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a])
I20250112 02:52:32.854395 1706 tablet_loader.cc:96] loaded metadata for tablet f05c8426ea3d4fe89fbe6c88c688019f (table default.Table_952b91d2b0b8435886e555f12629dc80 [id=ed2c162c64f74c9f889789b3abe7c041])
I20250112 02:52:32.854940 1706 tablet_loader.cc:96] loaded metadata for tablet f0dc78d722ac4687bfb3dce94d6db7f0 (table default.Table_0e7ff0c4e04e4b23a8baf069f96ac465 [id=d98e9e3bcb104f48a5102dea02c73c95])
I20250112 02:52:32.855206 1181 ts_tablet_manager.cc:1428] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250112 02:52:32.855865 1857 raft_consensus.cc:491] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.856004 1857 raft_consensus.cc:513] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.856518 1857 leader_election.cc:290] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:32.856791 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ee98f055024429eafaa29bb961cab87" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:32.856767 1181 tablet_bootstrap.cc:492] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:32.857187 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ee98f055024429eafaa29bb961cab87" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:32.857259 854 leader_election.cc:343] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.857614 1718 raft_consensus.cc:513] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:32.857781 855 leader_election.cc:343] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.857932 855 leader_election.cc:304] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:32.858206 1857 raft_consensus.cc:2743] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:32.858577 1718 leader_election.cc:290] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:32.858597 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61593969d2234ce3a6089027b067d922" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:32.858897 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61593969d2234ce3a6089027b067d922" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:32.859059 714 leader_election.cc:343] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:32.859408 714 leader_election.cc:343] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.859894 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61593969d2234ce3a6089027b067d922" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:32.860154 919 raft_consensus.cc:3054] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.862901 919 raft_consensus.cc:2463] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:32.863409 713 leader_election.cc:304] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:32.863826 1706 tablet_loader.cc:96] loaded metadata for tablet f31b777e7e0f43a28839d8c48c6a6590 (table default.Table_5fb592e41cc4406aa27546e7b41fdb88 [id=6402abce8d1d45c182cc989490c6a5f0])
I20250112 02:52:32.864063 1718 raft_consensus.cc:2798] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:32.864387 1706 tablet_loader.cc:96] loaded metadata for tablet f69b7a0f52304ad9951a07b3ab0372e7 (table default.Table_73e52096e12e47bab61cbfd2c72c70b1 [id=5424d9b98fae48949ed175c5498d24e1])
I20250112 02:52:32.864583 1718 raft_consensus.cc:695] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:32.864996 1718 consensus_queue.cc:237] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.865293 1706 tablet_loader.cc:96] loaded metadata for tablet f8c01b80fc8242d98fdfbf767212144d (table default.Table_63649e9607394f4bb72a3dd6e6d0dd69 [id=9807191b6e5c4d478cd47a54464fb9ce])
I20250112 02:52:32.865862 1706 tablet_loader.cc:96] loaded metadata for tablet f924459d1150472aafb325a11825db23 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1])
I20250112 02:52:32.871014 1173 tablet_bootstrap.cc:492] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:32.871182 1173 ts_tablet_manager.cc:1397] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.001s
I20250112 02:52:32.871866 1177 tablet_bootstrap.cc:492] T 4a57eb366f594f93a488c8250e195014 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:32.871896 1173 raft_consensus.cc:357] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.872056 1177 ts_tablet_manager.cc:1397] T 4a57eb366f594f93a488c8250e195014 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.162s user 0.026s sys 0.017s
I20250112 02:52:32.872128 1173 raft_consensus.cc:383] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.872196 1173 raft_consensus.cc:738] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.872443 1173 consensus_queue.cc:260] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.873380 1173 ts_tablet_manager.cc:1428] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:32.873335 1177 raft_consensus.cc:357] T 4a57eb366f594f93a488c8250e195014 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.873595 1177 raft_consensus.cc:383] T 4a57eb366f594f93a488c8250e195014 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.873723 1177 raft_consensus.cc:738] T 4a57eb366f594f93a488c8250e195014 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.873840 1181 tablet_bootstrap.cc:654] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.873960 1173 tablet_bootstrap.cc:492] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:32.874006 1177 consensus_queue.cc:260] T 4a57eb366f594f93a488c8250e195014 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.874716 1177 ts_tablet_manager.cc:1428] T 4a57eb366f594f93a488c8250e195014 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:32.875269 1177 tablet_bootstrap.cc:492] T 02d0f67141dc46b6a87ca0ad080d5bfe P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:32.875303 1706 tablet_loader.cc:96] loaded metadata for tablet fbe9e9624ede40c29ea852b2b5494965 (table default.Table_31424b670f274fd3abd17eb8451ba95c [id=ebc9aae273fb4bcd8bd904f2179f1dd4])
I20250112 02:52:32.875711 1706 tablet_loader.cc:96] loaded metadata for tablet fccff64af8ca4400afc116db98918119 (table default.Table_15bcb2c0f2a74458a1bed231fd3437fc [id=ca9c8281bf254f248076cb600e7c4f87])
I20250112 02:52:32.876103 1706 tablet_loader.cc:96] loaded metadata for tablet fd7394b1878447a189ce45957f10e4df (table default.Table_3d820aeff9534cc98f5722887aed5300 [id=08a760dcd20642fb869437d4ccc3f333])
I20250112 02:52:32.876479 1706 tablet_loader.cc:96] loaded metadata for tablet fd81852962fb43449e35ef45dc1daaad (table default.Table_d45a3d1878fc47d5917bf66c670c5578 [id=8a4387d6a5084b85a8f28cb99798cec9])
I20250112 02:52:32.877096 1706 tablet_loader.cc:96] loaded metadata for tablet fea2ef4e9a63491db850bb5393938017 (table default.Table_9f651533b2a6469aa03a744720bdc14a [id=68e2f0649cb9417e89b8c6ba0d723d32])
I20250112 02:52:32.877846 1173 tablet_bootstrap.cc:654] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.878054 1177 tablet_bootstrap.cc:654] T 02d0f67141dc46b6a87ca0ad080d5bfe P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.881330 1706 tablet_loader.cc:96] loaded metadata for tablet ff1b3ee229c445b19405cf4a19cf8003 (table default.Table_6b8cb70a5e5b4519af5d247f6b5bb8b0 [id=a28f5cfbcfe64768b40477ffc2053e27])
W20250112 02:52:32.885326 1706 catalog_manager.cc:1477] Time spent T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Loading table and tablet metadata into memory: real 1.270s user 0.150s sys 0.061s
I20250112 02:52:32.885413 1706 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 02:52:32.885382 464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd25f03f25df49178a5c53c636ecf304" candidate_term: 5 candidate_status { last_received { term: 4 index: 515 } } ignore_live_leader: false dest_uuid: "436220c23fa948efb5b6ca15875d8b0e" is_pre_election: true
I20250112 02:52:32.885978 1357 leader_election.cc:304] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd25f03f25df49178a5c53c636ecf304; no voters: 436220c23fa948efb5b6ca15875d8b0e, f4a49a5756b143fcb767c3f41463f8d9
I20250112 02:52:32.886456 1929 raft_consensus.cc:2743] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250112 02:52:32.886880 455 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:40610 (request call id 42) took 1221 ms. Trace:
I20250112 02:52:32.887012 455 rpcz_store.cc:276] 0112 02:52:31.665474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:31.668520 (+ 3046us) service_pool.cc:225] Handling call
0112 02:52:32.886843 (+1218323us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:52:32.893970 1173 tablet_bootstrap.cc:492] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:32.894119 1173 ts_tablet_manager.cc:1397] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.003s
I20250112 02:52:32.894915 1173 raft_consensus.cc:357] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.895133 1173 raft_consensus.cc:383] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.895232 1173 raft_consensus.cc:738] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.895489 1173 consensus_queue.cc:260] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.896098 1173 ts_tablet_manager.cc:1428] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 02:52:32.896669 1173 tablet_bootstrap.cc:492] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:32.898947 1173 tablet_bootstrap.cc:654] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.906903 1173 tablet_bootstrap.cc:492] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:32.908102 1706 catalog_manager.cc:1260] Loaded cluster ID: 273ff2f1d16946fcac5345b54ec7471c
I20250112 02:52:32.908310 1173 ts_tablet_manager.cc:1397] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20250112 02:52:32.909037 1173 raft_consensus.cc:357] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.909286 1706 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 02:52:32.909765 1173 raft_consensus.cc:383] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.909886 1173 raft_consensus.cc:738] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.911077 1173 consensus_queue.cc:260] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.911947 1177 tablet_bootstrap.cc:492] T 02d0f67141dc46b6a87ca0ad080d5bfe P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:32.912114 1177 ts_tablet_manager.cc:1397] T 02d0f67141dc46b6a87ca0ad080d5bfe P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.002s
I20250112 02:52:32.912902 1173 ts_tablet_manager.cc:1428] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 02:52:32.912827 1177 raft_consensus.cc:357] T 02d0f67141dc46b6a87ca0ad080d5bfe P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.913033 1177 raft_consensus.cc:383] T 02d0f67141dc46b6a87ca0ad080d5bfe P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.913100 1177 raft_consensus.cc:738] T 02d0f67141dc46b6a87ca0ad080d5bfe P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.913293 1177 consensus_queue.cc:260] T 02d0f67141dc46b6a87ca0ad080d5bfe P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:32.913770 1177 ts_tablet_manager.cc:1428] T 02d0f67141dc46b6a87ca0ad080d5bfe P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:32.914108 1177 tablet_bootstrap.cc:492] T daef3a58605e40978f7389698b22265d P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:32.916818 1173 tablet_bootstrap.cc:492] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:32.918661 1877 raft_consensus.cc:491] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.918812 1877 raft_consensus.cc:513] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.919371 1173 tablet_bootstrap.cc:654] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.920686 1181 tablet_bootstrap.cc:492] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:32.920827 1181 ts_tablet_manager.cc:1397] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.002s
I20250112 02:52:32.921526 1181 raft_consensus.cc:357] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.921747 1181 raft_consensus.cc:383] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.921821 1181 raft_consensus.cc:738] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.922040 1181 consensus_queue.cc:260] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.922334 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79c6bdd89f5a44efb56d65ccb724308b" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:32.922535 919 raft_consensus.cc:2463] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:32.922859 1706 catalog_manager.cc:1505] Loading token signing keys...
I20250112 02:52:32.923388 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79c6bdd89f5a44efb56d65ccb724308b" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:32.923509 989 leader_election.cc:304] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:32.923421 1877 leader_election.cc:290] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:32.923918 1866 raft_consensus.cc:2798] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:32.924156 1866 raft_consensus.cc:491] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:32.924242 1866 raft_consensus.cc:3054] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.925626 1177 tablet_bootstrap.cc:654] T daef3a58605e40978f7389698b22265d P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:32.929994 991 leader_election.cc:343] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.930660 1866 raft_consensus.cc:513] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.931929 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79c6bdd89f5a44efb56d65ccb724308b" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:32.931972 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79c6bdd89f5a44efb56d65ccb724308b" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:32.932134 919 raft_consensus.cc:3054] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.932715 1706 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Loaded TSK: 0
I20250112 02:52:32.933483 1181 ts_tablet_manager.cc:1428] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250112 02:52:32.934648 919 raft_consensus.cc:2463] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
W20250112 02:52:32.934983 991 leader_election.cc:343] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:32.935330 989 leader_election.cc:304] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:32.935613 1877 raft_consensus.cc:2798] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:32.935736 1877 raft_consensus.cc:695] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:32.935952 1877 consensus_queue.cc:237] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.936902 1866 leader_election.cc:290] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:32.941291 1181 tablet_bootstrap.cc:492] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:32.945890 1706 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 02:52:32.949774 1181 tablet_bootstrap.cc:654] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.961796 1177 tablet_bootstrap.cc:492] T daef3a58605e40978f7389698b22265d P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:32.961947 1177 ts_tablet_manager.cc:1397] T daef3a58605e40978f7389698b22265d P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.004s
I20250112 02:52:32.962646 1177 raft_consensus.cc:357] T daef3a58605e40978f7389698b22265d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.962855 1177 raft_consensus.cc:383] T daef3a58605e40978f7389698b22265d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.962970 1177 raft_consensus.cc:738] T daef3a58605e40978f7389698b22265d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.963186 1177 consensus_queue.cc:260] T daef3a58605e40978f7389698b22265d P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.963483 1173 tablet_bootstrap.cc:492] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:32.963622 1173 ts_tablet_manager.cc:1397] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.000s
I20250112 02:52:32.963727 1177 ts_tablet_manager.cc:1428] T daef3a58605e40978f7389698b22265d P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:32.964035 1177 tablet_bootstrap.cc:492] T 79c6bdd89f5a44efb56d65ccb724308b P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:32.964262 1173 raft_consensus.cc:357] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.964434 1173 raft_consensus.cc:383] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.964501 1173 raft_consensus.cc:738] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.964636 1877 consensus_queue.cc:1035] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:52:32.966236 1173 consensus_queue.cc:260] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.966799 1173 ts_tablet_manager.cc:1428] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 02:52:32.967065 1173 tablet_bootstrap.cc:492] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:32.968611 1173 tablet_bootstrap.cc:654] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.969879 1177 tablet_bootstrap.cc:654] T 79c6bdd89f5a44efb56d65ccb724308b P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:32.969813 454 catalog_manager.cc:5526] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:32.970767 454 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 79c6bdd89f5a44efb56d65ccb724308b (table default.Table_ad69cfacaf7b4050ab2d3d6ed237e7d0 [id=9e72fad141db43c09c7e4692769b6eba]). Expected version 1 got 0
I20250112 02:52:32.978601 1181 tablet_bootstrap.cc:492] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:32.978788 1181 ts_tablet_manager.cc:1397] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.004s
I20250112 02:52:32.984596 1181 raft_consensus.cc:357] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.984812 1181 raft_consensus.cc:383] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.984916 1181 raft_consensus.cc:738] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.985208 1181 consensus_queue.cc:260] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:32.986049 989 consensus_peers.cc:487] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42cb7e7a41634cf1bdb978ac9d2befcd. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:32.992372 1718 raft_consensus.cc:491] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:32.992547 1718 raft_consensus.cc:513] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.993217 1718 leader_election.cc:290] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:32.993247 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:32.993314 1177 tablet_bootstrap.cc:492] T 79c6bdd89f5a44efb56d65ccb724308b P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:32.993448 1177 ts_tablet_manager.cc:1397] T 79c6bdd89f5a44efb56d65ccb724308b P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.007s
I20250112 02:52:32.993937 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85300a9aa1c4c50a3957a004bea7933" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
W20250112 02:52:32.994099 455 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 4d25815a5df94e099e7481c48c3da793 consensus state has no leader
I20250112 02:52:32.994203 919 raft_consensus.cc:2463] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:32.994860 713 leader_election.cc:304] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:32.994781 1177 raft_consensus.cc:357] T 79c6bdd89f5a44efb56d65ccb724308b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.995069 1177 raft_consensus.cc:383] T 79c6bdd89f5a44efb56d65ccb724308b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:32.995182 1177 raft_consensus.cc:738] T 79c6bdd89f5a44efb56d65ccb724308b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:32.995405 1177 consensus_queue.cc:260] T 79c6bdd89f5a44efb56d65ccb724308b P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.995728 1609 raft_consensus.cc:2798] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:32.995815 1609 raft_consensus.cc:491] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:32.995894 1609 raft_consensus.cc:3054] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:32.997491 826 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:32.998353 1609 raft_consensus.cc:513] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:32.998895 1609 leader_election.cc:290] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:32.999773 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85300a9aa1c4c50a3957a004bea7933" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:32.999982 919 raft_consensus.cc:3054] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.002681 919 raft_consensus.cc:2463] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:33.003181 713 leader_election.cc:304] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:33.003710 1609 raft_consensus.cc:2798] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:33.003878 1609 raft_consensus.cc:695] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:33.004155 1609 consensus_queue.cc:237] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.006014 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85300a9aa1c4c50a3957a004bea7933" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:33.006203 1181 ts_tablet_manager.cc:1428] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.027s user 0.001s sys 0.002s
I20250112 02:52:33.006220 1054 raft_consensus.cc:3054] T a85300a9aa1c4c50a3957a004bea7933 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.009002 965 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:33.009375 1054 raft_consensus.cc:2463] T a85300a9aa1c4c50a3957a004bea7933 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:33.010836 1054 raft_consensus.cc:3054] T 7ea075826b16475ca474e284fc6bc545 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.014302 1718 consensus_queue.cc:1035] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:33.015967 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85300a9aa1c4c50a3957a004bea7933" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.016436 1181 tablet_bootstrap.cc:492] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:33.016577 1055 raft_consensus.cc:2371] T a85300a9aa1c4c50a3957a004bea7933 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1. Re-sending same reply.
I20250112 02:52:33.016346 1177 ts_tablet_manager.cc:1428] T 79c6bdd89f5a44efb56d65ccb724308b P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20250112 02:52:33.017683 1177 tablet_bootstrap.cc:492] T b2ccb88108f24e36ba4b7ebf36398950 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:33.026494 1173 tablet_bootstrap.cc:492] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:33.026687 1173 ts_tablet_manager.cc:1397] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.060s user 0.015s sys 0.001s
I20250112 02:52:33.027498 1173 raft_consensus.cc:357] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.027724 1173 raft_consensus.cc:383] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.027802 1173 raft_consensus.cc:738] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.028029 1173 consensus_queue.cc:260] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.028501 1173 ts_tablet_manager.cc:1428] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 02:52:33.028759 1173 tablet_bootstrap.cc:492] T b2ccb88108f24e36ba4b7ebf36398950 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:33.029898 1181 tablet_bootstrap.cc:654] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.030092 1173 tablet_bootstrap.cc:654] T b2ccb88108f24e36ba4b7ebf36398950 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.032374 1101 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
W20250112 02:52:33.038036 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4d25815a5df94e099e7481c48c3da793 consensus state has no leader
I20250112 02:52:33.043680 1718 raft_consensus.cc:491] T 79c6bdd89f5a44efb56d65ccb724308b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.043828 1718 raft_consensus.cc:513] T 79c6bdd89f5a44efb56d65ccb724308b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.044366 1718 leader_election.cc:290] T 79c6bdd89f5a44efb56d65ccb724308b P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.044993 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79c6bdd89f5a44efb56d65ccb724308b" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:33.045276 919 raft_consensus.cc:2388] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 289afc116aa74408aa5bff74e11c50d8 in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
I20250112 02:52:33.046747 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79c6bdd89f5a44efb56d65ccb724308b" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.046773 1602 consensus_queue.cc:1035] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:33.047729 714 leader_election.cc:304] T 79c6bdd89f5a44efb56d65ccb724308b P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:33.048208 1611 raft_consensus.cc:3054] T 79c6bdd89f5a44efb56d65ccb724308b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.050715 1611 raft_consensus.cc:2743] T 79c6bdd89f5a44efb56d65ccb724308b P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:33.051316 1181 tablet_bootstrap.cc:492] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:33.051483 1181 ts_tablet_manager.cc:1397] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.001s
I20250112 02:52:33.051764 1611 consensus_queue.cc:1035] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:33.052269 1181 raft_consensus.cc:357] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.052477 1181 raft_consensus.cc:383] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.052587 1181 raft_consensus.cc:738] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.052834 1181 consensus_queue.cc:260] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.053344 1181 ts_tablet_manager.cc:1428] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:33.053622 1181 tablet_bootstrap.cc:492] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:33.055882 1181 tablet_bootstrap.cc:654] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.076159 1181 tablet_bootstrap.cc:492] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:33.076306 1181 ts_tablet_manager.cc:1397] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20250112 02:52:33.076637 451 master_service.cc:946] ReplaceTablet: received request to replace tablet f05c8426ea3d4fe89fbe6c88c688019f from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:33.077299 1181 raft_consensus.cc:357] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.077549 1181 raft_consensus.cc:383] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.077626 1181 raft_consensus.cc:738] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.077862 1181 consensus_queue.cc:260] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.078521 1181 ts_tablet_manager.cc:1428] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:33.078804 1181 tablet_bootstrap.cc:492] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:33.078908 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_f7421d0779924239a3b493954ef748b7" } new_table_name: "default.Table_4fc60c2b6a554573bcca21da7a197875" modify_external_catalogs: true
I20250112 02:52:33.079073 450 master_service.cc:946] ReplaceTablet: received request to replace tablet afa2f8aabe5e42ccbc15b193fdc03563 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:33.079842 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_2b5aeec3b0bf40aea08010f65e422c2c" } new_table_name: "default.Table_ecbd292479064383a2b82283bbcfc614" modify_external_catalogs: true
I20250112 02:52:33.080539 448 master_service.cc:946] ReplaceTablet: received request to replace tablet b2ccb88108f24e36ba4b7ebf36398950 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:33.080735 1181 tablet_bootstrap.cc:654] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.081940 446 master_service.cc:946] ReplaceTablet: received request to replace tablet cb724feccc7f4fbc846fb6fa61be3385 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:33.081805 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_104ccd5f390f47a5878c67ea6de9d554"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:33.082729 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_104ccd5f390f47a5878c67ea6de9d554 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:33.085906 1877 raft_consensus.cc:491] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.086107 1877 raft_consensus.cc:513] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.086728 1877 leader_election.cc:290] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.086627 1718 consensus_queue.cc:1035] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:33.088088 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd48d0e68014e1aa0c44620527baec9" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:33.088897 991 leader_election.cc:343] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.089588 1181 tablet_bootstrap.cc:492] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:33.089748 1181 ts_tablet_manager.cc:1397] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20250112 02:52:33.090564 1181 raft_consensus.cc:357] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.090775 1181 raft_consensus.cc:383] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.090878 1181 raft_consensus.cc:738] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.091118 1181 consensus_queue.cc:260] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.091730 1181 ts_tablet_manager.cc:1428] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:33.092034 1181 tablet_bootstrap.cc:492] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:33.096735 1181 tablet_bootstrap.cc:654] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.102519 1173 tablet_bootstrap.cc:492] T b2ccb88108f24e36ba4b7ebf36398950 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:33.102802 1173 ts_tablet_manager.cc:1397] T b2ccb88108f24e36ba4b7ebf36398950 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.074s user 0.000s sys 0.004s
I20250112 02:52:33.104367 1173 raft_consensus.cc:357] T b2ccb88108f24e36ba4b7ebf36398950 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:33.104760 1173 raft_consensus.cc:383] T b2ccb88108f24e36ba4b7ebf36398950 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.104991 1173 raft_consensus.cc:738] T b2ccb88108f24e36ba4b7ebf36398950 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.105459 1173 consensus_queue.cc:260] T b2ccb88108f24e36ba4b7ebf36398950 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:33.112375 1181 tablet_bootstrap.cc:492] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:33.112515 1181 ts_tablet_manager.cc:1397] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.001s
I20250112 02:52:33.113297 1181 raft_consensus.cc:357] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:33.113520 1181 raft_consensus.cc:383] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.113591 1181 raft_consensus.cc:738] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.113351 1173 ts_tablet_manager.cc:1428] T b2ccb88108f24e36ba4b7ebf36398950 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20250112 02:52:33.113839 1181 consensus_queue.cc:260] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:33.114372 1181 ts_tablet_manager.cc:1428] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:33.114616 1181 tablet_bootstrap.cc:492] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:33.117565 1181 tablet_bootstrap.cc:654] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.118211 1173 tablet_bootstrap.cc:492] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:33.134204 760 tablet_service.cc:1514] Processing DeleteTablet for tablet daef3a58605e40978f7389698b22265d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:33.140300 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet f05c8426ea3d4fe89fbe6c88c688019f
I20250112 02:52:33.141258 451 catalog_manager.cc:6596] ReplaceTablet: tablet f05c8426ea3d4fe89fbe6c88c688019f deleted and replaced by tablet c5079f87652a4ea58b366d1f9448fd4e
I20250112 02:52:33.141366 744 tablet_service.cc:1514] Processing DeleteTablet for tablet f05c8426ea3d4fe89fbe6c88c688019f with delete_type TABLET_DATA_DELETED (replaced by tablet c5079f87652a4ea58b366d1f9448fd4e) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:33.141430 1857 raft_consensus.cc:491] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.141579 1857 raft_consensus.cc:513] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.142210 1857 leader_election.cc:290] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:33.143000 1173 tablet_bootstrap.cc:654] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.142959 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12c5d58e6c246c983e8a533906ce77d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:33.144132 855 leader_election.cc:343] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.154610 1857 raft_consensus.cc:491] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.154747 1857 raft_consensus.cc:513] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.155372 1857 leader_election.cc:290] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:33.155969 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b184806b4884cd3b0a002905a1e57af" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:33.156491 855 leader_election.cc:343] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.157430 1177 tablet_bootstrap.cc:700] T b2ccb88108f24e36ba4b7ebf36398950 P 289afc116aa74408aa5bff74e11c50d8: Time spent opening tablet: real 0.138s user 0.002s sys 0.000s
I20250112 02:52:33.157776 1177 tablet_bootstrap.cc:654] T b2ccb88108f24e36ba4b7ebf36398950 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.163888 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_e80769df10444dd5ae4bc23a8c6cc09e" } modify_external_catalogs: true
I20250112 02:52:33.164160 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_e80769df10444dd5ae4bc23a8c6cc09e" } modify_external_catalogs: true
I20250112 02:52:33.166059 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet b2ccb88108f24e36ba4b7ebf36398950
I20250112 02:52:33.166349 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_a5f330af613f4e5e9ecafdfe58eb0f1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:33.166585 448 catalog_manager.cc:6596] ReplaceTablet: tablet b2ccb88108f24e36ba4b7ebf36398950 deleted and replaced by tablet f3771166b77e4f90aaa116ab1a9dc299
I20250112 02:52:33.166957 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_8f2b912f93d84320b0f7e127c0543af6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:33.167416 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f2b912f93d84320b0f7e127c0543af6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:33.169517 744 tablet_service.cc:1514] Processing DeleteTablet for tablet b2ccb88108f24e36ba4b7ebf36398950 with delete_type TABLET_DATA_DELETED (replaced by tablet f3771166b77e4f90aaa116ab1a9dc299) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:33.170243 1611 consensus_queue.cc:1035] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:33.172356 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_613140584042445b9d71a3d05cee62c1" } new_table_name: "default.Table_5d473cabff2e46e0b531fbce28d9d558" modify_external_catalogs: true
W20250112 02:52:33.166967 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5f330af613f4e5e9ecafdfe58eb0f1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:33.177318 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd48d0e68014e1aa0c44620527baec9" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:33.177484 918 raft_consensus.cc:2463] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:33.179812 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_dd77b8845fdd46b7a2debbadaf553f75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:33.180291 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd77b8845fdd46b7a2debbadaf553f75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:33.185582 2003 tablet_replica.cc:331] T daef3a58605e40978f7389698b22265d P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:33.192549 2003 raft_consensus.cc:2238] T daef3a58605e40978f7389698b22265d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:33.196187 1877 raft_consensus.cc:491] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.196331 1877 raft_consensus.cc:513] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.196914 1877 leader_election.cc:290] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.198057 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12c5d58e6c246c983e8a533906ce77d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.198715 2003 raft_consensus.cc:2267] T daef3a58605e40978f7389698b22265d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:33.199867 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a57eb366f594f93a488c8250e195014" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.208626 775 raft_consensus.cc:2463] T 4a57eb366f594f93a488c8250e195014 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:33.210469 2003 ts_tablet_manager.cc:1905] T daef3a58605e40978f7389698b22265d P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:33.215657 1857 raft_consensus.cc:491] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.215826 1857 raft_consensus.cc:513] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.216406 1857 leader_election.cc:290] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
W20250112 02:52:33.216785 854 leader_election.cc:343] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.216914 854 leader_election.cc:304] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:33.217299 1859 raft_consensus.cc:2743] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:33.217547 991 leader_election.cc:304] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:33.217834 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b184806b4884cd3b0a002905a1e57af" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.218127 1877 raft_consensus.cc:2798] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:33.218221 1877 raft_consensus.cc:491] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:33.218302 1877 raft_consensus.cc:3054] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.218933 989 leader_election.cc:304] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:33.219164 1859 raft_consensus.cc:491] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.219264 1859 raft_consensus.cc:513] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.219722 1859 leader_election.cc:290] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:33.220244 1866 raft_consensus.cc:2798] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:33.220324 1866 raft_consensus.cc:491] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:33.220424 1866 raft_consensus.cc:3054] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:33.221841 989 consensus_peers.cc:487] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c7c530aa0aa64e47ae27d61f341d2bd9. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:33.222066 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a57eb366f594f93a488c8250e195014" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:33.222249 919 raft_consensus.cc:2463] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:33.212380 1177 tablet_bootstrap.cc:492] T b2ccb88108f24e36ba4b7ebf36398950 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:33.225603 1877 raft_consensus.cc:513] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.236447 1877 leader_election.cc:290] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.227847 1866 raft_consensus.cc:513] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.237180 1866 leader_election.cc:290] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:33.237543 854 leader_election.cc:343] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.237670 854 leader_election.cc:304] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:33.238013 1859 raft_consensus.cc:2743] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:33.238557 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a57eb366f594f93a488c8250e195014" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:33.238751 919 raft_consensus.cc:3054] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.238804 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd48d0e68014e1aa0c44620527baec9" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:33.238972 918 raft_consensus.cc:3054] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.241588 919 raft_consensus.cc:2463] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:33.241649 918 raft_consensus.cc:2463] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:33.242712 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31b777e7e0f43a28839d8c48c6a6590" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.243755 1181 tablet_bootstrap.cc:492] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:33.243876 1181 ts_tablet_manager.cc:1397] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.129s user 0.004s sys 0.028s
I20250112 02:52:33.244571 1181 raft_consensus.cc:357] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.244750 1181 raft_consensus.cc:383] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.244818 1181 raft_consensus.cc:738] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.245010 1181 consensus_queue.cc:260] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.245566 1181 ts_tablet_manager.cc:1428] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:33.246503 1181 tablet_bootstrap.cc:492] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:33.248109 1859 raft_consensus.cc:491] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.248268 1859 raft_consensus.cc:513] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:33.248716 1181 tablet_bootstrap.cc:654] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.248901 1859 leader_election.cc:290] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:33.259546 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31b777e7e0f43a28839d8c48c6a6590" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.260804 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_787c491d6a594a27b46814669b3ccc2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:33.261349 855 leader_election.cc:343] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:33.261507 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_787c491d6a594a27b46814669b3ccc2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:33.263129 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416fb6820cd54e118950c68d381f51b2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.263540 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd48d0e68014e1aa0c44620527baec9" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
W20250112 02:52:33.264096 855 leader_election.cc:343] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.263893 1866 consensus_queue.cc:1035] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250112 02:52:33.264247 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416fb6820cd54e118950c68d381f51b2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.264899 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a57eb366f594f93a488c8250e195014" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:33.265065 779 raft_consensus.cc:3054] T 4a57eb366f594f93a488c8250e195014 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.266302 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_28b2d14acaa04d2a90164800ca8ced3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:33.266808 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28b2d14acaa04d2a90164800ca8ced3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:33.267947 1181 tablet_bootstrap.cc:492] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:33.268083 1181 ts_tablet_manager.cc:1397] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.000s
I20250112 02:52:33.268451 1602 consensus_queue.cc:1035] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250112 02:52:33.268841 1181 raft_consensus.cc:357] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.269057 1181 raft_consensus.cc:383] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.269116 1181 raft_consensus.cc:738] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.270236 1181 consensus_queue.cc:260] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.271118 1181 ts_tablet_manager.cc:1428] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:33.271230 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_fbc52174b22b4aafa4ede40120b90daf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:33.271911 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fbc52174b22b4aafa4ede40120b90daf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:33.272004 1181 tablet_bootstrap.cc:492] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:33.274217 779 raft_consensus.cc:2463] T 4a57eb366f594f93a488c8250e195014 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:33.274678 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7394b1878447a189ce45957f10e4df" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.277439 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_24fca5290fd54774bee413c9d84a997d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:33.277897 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24fca5290fd54774bee413c9d84a997d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:33.277878 1609 consensus_queue.cc:1035] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.055s
I20250112 02:52:33.278221 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_9c10574f7c7d435890eb9a24af23bcb9" } modify_external_catalogs: true
I20250112 02:52:33.278406 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_9c10574f7c7d435890eb9a24af23bcb9" } modify_external_catalogs: true
I20250112 02:52:33.283113 1734 consensus_queue.cc:1035] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:52:33.275271 855 leader_election.cc:343] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.284839 1181 tablet_bootstrap.cc:654] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.295979 1181 tablet_bootstrap.cc:492] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:33.296109 1181 ts_tablet_manager.cc:1397] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20250112 02:52:33.296835 1181 raft_consensus.cc:357] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:33.297019 1181 raft_consensus.cc:383] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.297150 1181 raft_consensus.cc:738] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.301937 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_27f1bff714d947d5b3f42be6bfbe8be2" } new_table_name: "default.Table_c16c321e8c3741838921ff50f04b8261" modify_external_catalogs: true
I20250112 02:52:33.307272 1181 consensus_queue.cc:260] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:33.307735 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_af9d530db1774d9eafed6727be07aeaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:33.308725 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af9d530db1774d9eafed6727be07aeaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:33.313575 1177 ts_tablet_manager.cc:1397] T b2ccb88108f24e36ba4b7ebf36398950 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.211s user 0.003s sys 0.002s
I20250112 02:52:33.314332 1177 raft_consensus.cc:357] T b2ccb88108f24e36ba4b7ebf36398950 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:33.314632 1177 raft_consensus.cc:383] T b2ccb88108f24e36ba4b7ebf36398950 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.314742 1177 raft_consensus.cc:738] T b2ccb88108f24e36ba4b7ebf36398950 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.315214 1177 consensus_queue.cc:260] T b2ccb88108f24e36ba4b7ebf36398950 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:33.319128 989 leader_election.cc:304] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
W20250112 02:52:33.319581 991 leader_election.cc:343] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.319562 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_3321055e22c3410eb5e5fe8be62e004c" } new_table_name: "default.Table_00028054ce7c431b96d17d8f6f8856a7" modify_external_catalogs: true
W20250112 02:52:33.319947 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 50878us
I20250112 02:52:33.320252 991 leader_election.cc:304] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:33.320523 1866 raft_consensus.cc:2798] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:33.320648 1866 raft_consensus.cc:695] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:33.320881 1866 consensus_queue.cc:237] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.319345 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7394b1878447a189ce45957f10e4df" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.324623 1866 raft_consensus.cc:2798] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:33.324784 1866 raft_consensus.cc:695] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:33.325088 1866 consensus_queue.cc:237] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:33.327454 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 68063.4us
I20250112 02:52:33.328732 1866 raft_consensus.cc:491] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.328914 1866 raft_consensus.cc:513] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.330077 1866 leader_election.cc:290] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.330331 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b599e983ec04f53a440d1ab828d12ab" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:33.330896 991 leader_election.cc:343] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.350745 1173 tablet_bootstrap.cc:492] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:33.350960 1173 ts_tablet_manager.cc:1397] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.234s user 0.062s sys 0.078s
I20250112 02:52:33.352039 1173 raft_consensus.cc:357] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.352277 1173 raft_consensus.cc:383] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.352396 1173 raft_consensus.cc:738] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.352660 1173 consensus_queue.cc:260] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.353401 1173 ts_tablet_manager.cc:1428] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:33.353929 1177 ts_tablet_manager.cc:1428] T b2ccb88108f24e36ba4b7ebf36398950 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20250112 02:52:33.353693 1173 tablet_bootstrap.cc:492] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:33.354377 1177 tablet_bootstrap.cc:492] T 7a18132549374218893f35a4e4226b72 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:33.356019 1177 tablet_bootstrap.cc:654] T 7a18132549374218893f35a4e4226b72 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.356570 1173 tablet_bootstrap.cc:654] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.357280 1866 raft_consensus.cc:491] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.357422 1866 raft_consensus.cc:513] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:33.358214 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a18132549374218893f35a4e4226b72" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.358695 1866 leader_election.cc:290] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:33.358992 991 leader_election.cc:343] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.363814 779 raft_consensus.cc:3054] T 02d0f67141dc46b6a87ca0ad080d5bfe P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.368400 1877 consensus_queue.cc:1035] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:33.375859 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd48d0e68014e1aa0c44620527baec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:33.376427 1173 tablet_bootstrap.cc:492] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:33.376590 1173 ts_tablet_manager.cc:1397] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.013s
I20250112 02:52:33.389437 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet f05c8426ea3d4fe89fbe6c88c688019f with delete_type TABLET_DATA_DELETED (replaced by tablet c5079f87652a4ea58b366d1f9448fd4e) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:33.390461 1173 raft_consensus.cc:357] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.390595 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet b2ccb88108f24e36ba4b7ebf36398950 with delete_type TABLET_DATA_DELETED (replaced by tablet f3771166b77e4f90aaa116ab1a9dc299) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:33.390703 1173 raft_consensus.cc:383] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.390692 778 raft_consensus.cc:1270] T 79c6bdd89f5a44efb56d65ccb724308b P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Refusing update from remote peer f1e775296eb746db8b318f17d26820e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 02:52:33.390798 1173 raft_consensus.cc:738] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.391012 1173 consensus_queue.cc:260] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.391273 1866 consensus_queue.cc:1035] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:33.391623 1173 ts_tablet_manager.cc:1428] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.015s user 0.002s sys 0.001s
I20250112 02:52:33.395085 2024 tablet_replica.cc:331] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:33.395404 2024 raft_consensus.cc:2238] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:33.395787 2024 pending_rounds.cc:62] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6: Trying to abort 1 pending ops.
I20250112 02:52:33.395895 2024 pending_rounds.cc:69] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319847221846016 op_type: NO_OP noop_request { }
I20250112 02:52:33.396111 2024 raft_consensus.cc:2883] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:33.396260 2024 raft_consensus.cc:2267] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:33.397951 2024 ts_tablet_manager.cc:1905] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:33.398428 2028 tablet.cc:1721] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:33.411551 2029 tablet.cc:1721] T 79c6bdd89f5a44efb56d65ccb724308b P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:33.412845 1177 tablet_bootstrap.cc:492] T 7a18132549374218893f35a4e4226b72 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:33.423370 1177 ts_tablet_manager.cc:1397] T 7a18132549374218893f35a4e4226b72 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.000s
I20250112 02:52:33.424144 1177 raft_consensus.cc:357] T 7a18132549374218893f35a4e4226b72 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:33.424386 1177 raft_consensus.cc:383] T 7a18132549374218893f35a4e4226b72 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.424510 1177 raft_consensus.cc:738] T 7a18132549374218893f35a4e4226b72 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.424829 1177 consensus_queue.cc:260] T 7a18132549374218893f35a4e4226b72 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:33.429589 1177 ts_tablet_manager.cc:1428] T 7a18132549374218893f35a4e4226b72 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250112 02:52:33.429881 1177 tablet_bootstrap.cc:492] T 4b599e983ec04f53a440d1ab828d12ab P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:33.431599 1177 tablet_bootstrap.cc:654] T 4b599e983ec04f53a440d1ab828d12ab P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.410280 2024 ts_tablet_manager.cc:1918] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:33.443599 2024 log.cc:1198] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/5dd48d0e68014e1aa0c44620527baec9
I20250112 02:52:33.443996 2024 ts_tablet_manager.cc:1939] T 5dd48d0e68014e1aa0c44620527baec9 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:33.417775 1173 tablet_bootstrap.cc:492] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:33.444679 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 5dd48d0e68014e1aa0c44620527baec9 (table default.Table_66559fa571be4748b005f229c7089c09 [id=5431f5161ddf40d083b94b5fcb6b7561]) successfully deleted
I20250112 02:52:33.445616 1173 tablet_bootstrap.cc:654] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:33.449497 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f05c8426ea3d4fe89fbe6c88c688019f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f05c8426ea3d4fe89fbe6c88c688019f already in progress: creating tablet
I20250112 02:52:33.449882 2024 tablet_replica.cc:331] T b2ccb88108f24e36ba4b7ebf36398950 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:33.450088 2024 raft_consensus.cc:2238] T b2ccb88108f24e36ba4b7ebf36398950 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:33.450286 2024 raft_consensus.cc:2267] T b2ccb88108f24e36ba4b7ebf36398950 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:33.458441 1181 ts_tablet_manager.cc:1428] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.162s user 0.002s sys 0.000s
I20250112 02:52:33.458797 1181 tablet_bootstrap.cc:492] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:33.459084 892 tablet_service.cc:1514] Processing DeleteTablet for tablet f05c8426ea3d4fe89fbe6c88c688019f with delete_type TABLET_DATA_DELETED (replaced by tablet c5079f87652a4ea58b366d1f9448fd4e) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:33.459403 898 tablet_service.cc:1514] Processing DeleteTablet for tablet b2ccb88108f24e36ba4b7ebf36398950 with delete_type TABLET_DATA_DELETED (replaced by tablet f3771166b77e4f90aaa116ab1a9dc299) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:33.459985 1859 raft_consensus.cc:491] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 02:52:33.460158 854 leader_election.cc:343] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.461017 2003 ts_tablet_manager.cc:1918] T daef3a58605e40978f7389698b22265d P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:33.461160 2003 log.cc:1198] T daef3a58605e40978f7389698b22265d P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/daef3a58605e40978f7389698b22265d
I20250112 02:52:33.461090 1859 raft_consensus.cc:513] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.461585 2003 ts_tablet_manager.cc:1939] T daef3a58605e40978f7389698b22265d P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:33.461882 1859 leader_election.cc:290] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:33.462252 1859 raft_consensus.cc:491] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.462360 1859 raft_consensus.cc:513] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.462845 2003 tablet_replica.cc:331] T b2ccb88108f24e36ba4b7ebf36398950 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:33.462900 1859 leader_election.cc:290] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:33.463101 2003 raft_consensus.cc:2238] T b2ccb88108f24e36ba4b7ebf36398950 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:33.463438 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet daef3a58605e40978f7389698b22265d (table default.Table_8e00e020edc14465ad0e935be514d03d [id=cf8528b7c33a47ce984da87382bef6a5]) successfully deleted
W20250112 02:52:33.463743 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f05c8426ea3d4fe89fbe6c88c688019f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f05c8426ea3d4fe89fbe6c88c688019f already in progress: creating tablet
I20250112 02:52:33.464941 1181 tablet_bootstrap.cc:654] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.466883 2036 tablet_replica.cc:331] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:33.467790 854 leader_election.cc:304] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:33.468206 854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 140134us
I20250112 02:52:33.469877 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe614eaeb793473d9b2da206b7345562" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.470260 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e933b58dab0b45e8bf1c49eebb1c53e3" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.471530 1857 raft_consensus.cc:491] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.471733 1857 raft_consensus.cc:513] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.474004 1857 leader_election.cc:290] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:33.474141 1857 raft_consensus.cc:2743] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:33.474444 1857 raft_consensus.cc:491] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.474550 1857 raft_consensus.cc:513] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.475021 1857 leader_election.cc:290] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:33.475132 1857 raft_consensus.cc:491] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.475260 1857 raft_consensus.cc:513] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.475692 1857 leader_election.cc:290] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:33.475828 1857 raft_consensus.cc:491] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.475951 1857 raft_consensus.cc:513] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.476089 1859 raft_consensus.cc:491] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.476130 1177 tablet_bootstrap.cc:492] T 4b599e983ec04f53a440d1ab828d12ab P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:33.476259 1177 ts_tablet_manager.cc:1397] T 4b599e983ec04f53a440d1ab828d12ab P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
I20250112 02:52:33.476244 1859 raft_consensus.cc:513] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.476404 1857 leader_election.cc:290] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:33.476698 2036 raft_consensus.cc:2238] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:33.476728 1859 leader_election.cc:290] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:33.476892 2036 raft_consensus.cc:2267] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:33.476903 1177 raft_consensus.cc:357] T 4b599e983ec04f53a440d1ab828d12ab P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.477409 2036 ts_tablet_manager.cc:1905] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:33.478035 1857 raft_consensus.cc:491] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.478319 1857 raft_consensus.cc:513] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.478543 2035 raft_consensus.cc:491] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.478722 2035 raft_consensus.cc:513] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.478852 1857 leader_election.cc:290] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
W20250112 02:52:33.479048 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet f05c8426ea3d4fe89fbe6c88c688019f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f05c8426ea3d4fe89fbe6c88c688019f already in progress: creating tablet
I20250112 02:52:33.479384 2035 leader_election.cc:290] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:33.477101 1177 raft_consensus.cc:383] T 4b599e983ec04f53a440d1ab828d12ab P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.479537 1177 raft_consensus.cc:738] T 4b599e983ec04f53a440d1ab828d12ab P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.480144 1177 consensus_queue.cc:260] T 4b599e983ec04f53a440d1ab828d12ab P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.480458 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dda52f012794493a102ed1abc718f5b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.481029 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e933b58dab0b45e8bf1c49eebb1c53e3" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.481318 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87759e8b99b04e628badc8f60cb38bb3" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.481606 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2730634a5cfd4c2fb5c4e30100c556d4" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.482422 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac716c6da01941a989754aac1ef430d5" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.482767 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352c87f1dc724e7b94bc2e64abfce8d7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.484320 1177 ts_tablet_manager.cc:1428] T 4b599e983ec04f53a440d1ab828d12ab P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250112 02:52:33.484685 1177 tablet_bootstrap.cc:492] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:33.486364 1177 tablet_bootstrap.cc:654] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.487548 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9537fcd96eb41cc8f680075ec71137b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:33.492588 855 leader_election.cc:343] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:33.492872 855 leader_election.cc:343] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:33.493088 855 leader_election.cc:343] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:33.493260 855 leader_election.cc:343] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:33.493441 855 leader_election.cc:343] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:33.493897 855 leader_election.cc:343] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:33.494190 855 leader_election.cc:343] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.494663 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d25815a5df94e099e7481c48c3da793" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.495885 2024 ts_tablet_manager.cc:1905] T b2ccb88108f24e36ba4b7ebf36398950 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:33.501613 2036 ts_tablet_manager.cc:1918] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:33.502007 2036 log.cc:1198] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/b2ccb88108f24e36ba4b7ebf36398950
I20250112 02:52:33.503429 2036 ts_tablet_manager.cc:1939] T b2ccb88108f24e36ba4b7ebf36398950 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:33.503764 1727 raft_consensus.cc:491] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.503922 1727 raft_consensus.cc:513] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.504261 2003 raft_consensus.cc:2267] T b2ccb88108f24e36ba4b7ebf36398950 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:33.504575 1727 leader_election.cc:290] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.505159 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d69632845547f588662dd56c8c194e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:33.505620 991 leader_election.cc:343] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.506301 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 7e1954e7859f422f9690fe4a9384a8bb (table default.Table_829049fe22b74e62a87dd1bed65fb3fe [id=2e892eba68234ffca0be4ef3e49cf309]). Expected version 1 got 0
I20250112 02:52:33.506656 1173 tablet_bootstrap.cc:492] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:33.507210 1173 ts_tablet_manager.cc:1397] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.090s user 0.010s sys 0.008s
I20250112 02:52:33.507499 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:33.507846 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:33.508663 1173 raft_consensus.cc:357] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.508915 1173 raft_consensus.cc:383] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.509016 1173 raft_consensus.cc:738] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.509213 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:33.509354 1173 consensus_queue.cc:260] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.509750 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:33.511644 1727 consensus_queue.cc:1035] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:33.513233 2024 ts_tablet_manager.cc:1918] T b2ccb88108f24e36ba4b7ebf36398950 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:33.513870 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
W20250112 02:52:33.508989 855 leader_election.cc:343] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.519057 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe614eaeb793473d9b2da206b7345562" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:33.519683 855 leader_election.cc:343] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.521055 2032 raft_consensus.cc:491] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.521222 2032 raft_consensus.cc:513] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:33.522017 2032 leader_election.cc:290] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:33.516357 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet b2ccb88108f24e36ba4b7ebf36398950 (table default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6]) successfully deleted
I20250112 02:52:33.522832 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0c3191973a477e949fbf9dfcca963f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.522949 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
W20250112 02:52:33.523339 855 leader_election.cc:343] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.524524 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 9af55dab202444358ea1a0674d37946d (table default.Table_26dfe346379948b39f90b31653c2a463 [id=6c6f30e60e79451397841d090d11d8b7]). Expected version 1 got 0
I20250112 02:52:33.527415 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:33.527813 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:33.528354 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet f31b777e7e0f43a28839d8c48c6a6590 (table default.Table_5fb592e41cc4406aa27546e7b41fdb88 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 1 got 0
I20250112 02:52:33.531944 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 6117dc8b32a64535a2f4167a03bed0d2 (table default.Table_365739770a11434dbf260ea912e7ea81 [id=868adb627654450c936ef198e49d853b]). Expected version 1 got 0
I20250112 02:52:33.535820 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 8ee98f055024429eafaa29bb961cab87 (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 1 got 0
I20250112 02:52:33.536626 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 79c6bdd89f5a44efb56d65ccb724308b (table default.Table_ad69cfacaf7b4050ab2d3d6ed237e7d0 [id=9e72fad141db43c09c7e4692769b6eba]). Expected version 1 got 0
I20250112 02:52:33.515256 1173 ts_tablet_manager.cc:1428] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250112 02:52:33.537096 1173 tablet_bootstrap.cc:492] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:33.538806 1173 tablet_bootstrap.cc:654] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.540774 2003 ts_tablet_manager.cc:1905] T b2ccb88108f24e36ba4b7ebf36398950 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:33.544171 2024 log.cc:1198] T b2ccb88108f24e36ba4b7ebf36398950 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/b2ccb88108f24e36ba4b7ebf36398950
I20250112 02:52:33.544976 2024 ts_tablet_manager.cc:1939] T b2ccb88108f24e36ba4b7ebf36398950 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:33.545491 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 7ea075826b16475ca474e284fc6bc545 (table default.Table_009a6976f6db4f218e3c3373368a58d7 [id=c5005669bb5f4c029d673bd39cc1535d]). Expected version 1 got 0
I20250112 02:52:33.546702 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet b2ccb88108f24e36ba4b7ebf36398950 (table default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6]) successfully deleted
I20250112 02:52:33.546895 452 catalog_manager.cc:5526] T f8c01b80fc8242d98fdfbf767212144d P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } }
I20250112 02:52:33.548174 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet f8c01b80fc8242d98fdfbf767212144d (table default.Table_63649e9607394f4bb72a3dd6e6d0dd69 [id=9807191b6e5c4d478cd47a54464fb9ce]). Expected version 1 got 0
I20250112 02:52:33.548678 452 catalog_manager.cc:5526] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } }
I20250112 02:52:33.549232 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 2c03b5e612d64314b210c11686fdfa2a (table default.Table_4934f6022f2441eda88751236a5798be [id=5c4769bfcefd4fe0a0f3c2e315d50c32]). Expected version 6 got 0
I20250112 02:52:33.549492 452 catalog_manager.cc:5526] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:33.550362 1181 tablet_bootstrap.cc:492] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:33.550586 1181 ts_tablet_manager.cc:1397] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.092s user 0.005s sys 0.003s
I20250112 02:52:33.550626 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 23e7a2e88c374e949ced92c557225cbf (table default.Table_cd1f185d82b3474083246ec3702851fc [id=bfa9a504a8d34c86929efb0d15d049d0]). Expected version 3 got 0
I20250112 02:52:33.551563 1181 raft_consensus.cc:357] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.551863 1181 raft_consensus.cc:383] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.551955 1181 raft_consensus.cc:738] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.552234 1181 consensus_queue.cc:260] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.552896 1181 ts_tablet_manager.cc:1428] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250112 02:52:33.550952 452 catalog_manager.cc:5526] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } }
I20250112 02:52:33.553161 1181 tablet_bootstrap.cc:492] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:33.553311 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ab2e31a65294407bb62b13e92898a7d2 (table default.Table_3321055e22c3410eb5e5fe8be62e004c [id=f8cd4289f89a4a748c26a7a01236406e]). Expected version 2 got 0
I20250112 02:52:33.554525 1181 tablet_bootstrap.cc:654] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.555802 452 catalog_manager.cc:5526] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:33.556185 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c04b64bc2f8348a086ed3c4e59f09b16 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:33.556308 1173 tablet_bootstrap.cc:492] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:33.556443 1173 ts_tablet_manager.cc:1397] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.008s
I20250112 02:52:33.557997 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b599e983ec04f53a440d1ab828d12ab" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:33.558243 919 raft_consensus.cc:2463] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:33.558588 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a18132549374218893f35a4e4226b72" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:33.558751 919 raft_consensus.cc:2463] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:33.559242 919 raft_consensus.cc:1270] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Refusing update from remote peer f1e775296eb746db8b318f17d26820e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 02:52:33.561261 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d69632845547f588662dd56c8c194e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:33.561431 919 raft_consensus.cc:2463] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
W20250112 02:52:33.562075 854 leader_election.cc:343] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.562215 854 leader_election.cc:304] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:33.562582 854 leader_election.cc:343] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.562758 854 leader_election.cc:304] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:33.563243 2026 raft_consensus.cc:2743] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:33.563421 2026 raft_consensus.cc:2743] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:33.567799 854 leader_election.cc:343] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.568943 1177 tablet_bootstrap.cc:492] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:33.568928 854 leader_election.cc:304] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:33.569126 1177 ts_tablet_manager.cc:1397] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.084s user 0.014s sys 0.014s
W20250112 02:52:33.569344 854 leader_election.cc:343] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.569473 854 leader_election.cc:304] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:33.569964 1177 raft_consensus.cc:357] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.570222 1177 raft_consensus.cc:383] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.570323 1177 raft_consensus.cc:738] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.570590 1177 consensus_queue.cc:260] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.571733 1177 ts_tablet_manager.cc:1428] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:33.572026 1177 tablet_bootstrap.cc:492] T 88d69632845547f588662dd56c8c194e P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:33.574256 1177 tablet_bootstrap.cc:654] T 88d69632845547f588662dd56c8c194e P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.574394 2039 raft_consensus.cc:2743] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:33.575129 2039 raft_consensus.cc:2743] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:33.576504 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d25815a5df94e099e7481c48c3da793" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.576700 2038 raft_consensus.cc:491] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.576778 989 leader_election.cc:304] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:33.576853 2038 raft_consensus.cc:513] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.577441 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2730634a5cfd4c2fb5c4e30100c556d4" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.577811 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb76746b3a846c3aba531c04bf63966" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.577998 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac716c6da01941a989754aac1ef430d5" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:33.578225 855 leader_election.cc:343] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.578336 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352c87f1dc724e7b94bc2e64abfce8d7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.578414 2038 leader_election.cc:290] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:33.578629 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dda52f012794493a102ed1abc718f5b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.556604 452 catalog_manager.cc:5526] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:33.579658 2003 ts_tablet_manager.cc:1918] T b2ccb88108f24e36ba4b7ebf36398950 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:33.579799 2003 log.cc:1198] T b2ccb88108f24e36ba4b7ebf36398950 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/b2ccb88108f24e36ba4b7ebf36398950
I20250112 02:52:33.579808 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 7cd9985cd20148f195a857c478b0755f (table default.Table_9b65dd291c9f4fcdaa6d5e37f04d24ae [id=e63daf724d874c89badafb8d59653f74]). Expected version 1 got 0
I20250112 02:52:33.579991 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 003a3b4705e943c6a900762ec11ddd47 (table default.Table_743b3f7029534564b35c0c137b35fd75 [id=b2fa3922f6d447769a8179052f9a1861]). Expected version 1 got 0
I20250112 02:52:33.580117 2003 ts_tablet_manager.cc:1939] T b2ccb88108f24e36ba4b7ebf36398950 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:33.580154 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 82b351c563d749df806ab5d0331a4ef2 (table default.Table_4603068c01544e2e965dca5177e36b40 [id=8502f21e9b61436093db1241cfbb5ec9]). Expected version 1 got 0
I20250112 02:52:33.580425 452 catalog_manager.cc:5526] T 27ab58e45d0d44b283019240f44b66bf P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } }
I20250112 02:52:33.580829 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:33.581272 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet b2ccb88108f24e36ba4b7ebf36398950 (table default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6]) successfully deleted
I20250112 02:52:33.581895 1181 tablet_bootstrap.cc:492] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:33.581995 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:33.582057 1181 ts_tablet_manager.cc:1397] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.001s
I20250112 02:52:33.582230 452 catalog_manager.cc:5526] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } }
I20250112 02:52:33.578969 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9537fcd96eb41cc8f680075ec71137b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.583024 1181 raft_consensus.cc:357] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.583276 1181 raft_consensus.cc:383] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.583369 1181 raft_consensus.cc:738] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.583585 1181 consensus_queue.cc:260] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.583822 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 4916ae3917084cfe8c93240a8857be0e (table default.Table_69f363c4da234792886b544823b3214e [id=c7052afaf6284ef0972835c1f46cb829]). Expected version 1 got 0
I20250112 02:52:33.584158 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_feaebd2f4cbb47b6b24ff79614994d6f [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 1 got 0
I20250112 02:52:33.584733 1877 raft_consensus.cc:2798] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:33.584646 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87759e8b99b04e628badc8f60cb38bb3" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.585096 1877 raft_consensus.cc:491] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:33.585212 1877 raft_consensus.cc:3054] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.586089 1181 ts_tablet_manager.cc:1428] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250112 02:52:33.586324 1181 tablet_bootstrap.cc:492] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:33.586328 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:33.586670 452 catalog_manager.cc:5526] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } }
I20250112 02:52:33.583547 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0c3191973a477e949fbf9dfcca963f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.588362 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet d8eb0efc5d7f4c72bdfc7531a444d42a (table default.Table_d561e1740cc545dc8aa4babb76b45c37 [id=de27aeff6b3a49b3956498ea5015ff33]). Expected version 1 got 0
I20250112 02:52:33.588579 989 leader_election.cc:304] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:33.588868 1866 raft_consensus.cc:491] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.589004 1866 raft_consensus.cc:513] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.589347 1877 raft_consensus.cc:513] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.589592 1866 leader_election.cc:290] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.589926 1877 leader_election.cc:290] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.590169 1877 raft_consensus.cc:2798] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:33.590294 1877 raft_consensus.cc:491] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:33.590544 1877 raft_consensus.cc:3054] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.598404 1877 raft_consensus.cc:513] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:33.599620 1181 tablet_bootstrap.cc:654] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.600405 1877 leader_election.cc:290] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.601150 452 catalog_manager.cc:5526] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } }
I20250112 02:52:33.604636 1181 tablet_bootstrap.cc:492] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:33.604840 1181 ts_tablet_manager.cc:1397] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.002s
I20250112 02:52:33.604661 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 09d06e2e8c13464db8cf801a292bdc99 (table default.Table_0bdc5f49d4ff4d6fb1290bc2dc0f70d9 [id=e3bb489c53454222a79702b798c0c2f1]). Expected version 2 got 0
I20250112 02:52:33.605453 1181 raft_consensus.cc:357] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.605641 1181 raft_consensus.cc:383] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.605706 1181 raft_consensus.cc:738] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.605891 1181 consensus_queue.cc:260] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.605392 452 catalog_manager.cc:5526] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } }
I20250112 02:52:33.606685 1181 ts_tablet_manager.cc:1428] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:33.607134 452 catalog_manager.cc:5526] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } }
I20250112 02:52:33.607534 1181 tablet_bootstrap.cc:492] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:33.608147 2049 raft_consensus.cc:491] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.608552 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet abfaad67d5e34c0a8338dccc0ef05466 (table default.Table_045e5b104dc64217aa39c22534007cab [id=2c0e94e8c5774da1b8130c7651d94500]). Expected version 1 got 0
I20250112 02:52:33.609211 1181 tablet_bootstrap.cc:654] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.609650 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61593969d2234ce3a6089027b067d922" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.609987 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b599e983ec04f53a440d1ab828d12ab" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:33.610157 778 raft_consensus.cc:3054] T 4b599e983ec04f53a440d1ab828d12ab P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.609068 1866 consensus_queue.cc:1035] T 79c6bdd89f5a44efb56d65ccb724308b P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:33.610636 1866 consensus_queue.cc:1035] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:33.611027 1866 consensus_queue.cc:1035] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:33.609639 1173 raft_consensus.cc:357] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.611850 1173 raft_consensus.cc:383] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.611958 1173 raft_consensus.cc:738] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.612143 1173 consensus_queue.cc:260] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.612675 1173 ts_tablet_manager.cc:1428] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.056s user 0.000s sys 0.000s
I20250112 02:52:33.612898 778 raft_consensus.cc:2463] T 4b599e983ec04f53a440d1ab828d12ab P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:33.613142 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a18132549374218893f35a4e4226b72" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:33.613308 771 raft_consensus.cc:3054] T 7a18132549374218893f35a4e4226b72 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.613382 1173 tablet_bootstrap.cc:492] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:33.615250 1173 tablet_bootstrap.cc:654] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.616022 771 raft_consensus.cc:2463] T 7a18132549374218893f35a4e4226b72 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:33.627313 1181 tablet_bootstrap.cc:492] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:33.627503 1181 ts_tablet_manager.cc:1397] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.005s
I20250112 02:52:33.628173 1181 raft_consensus.cc:357] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.628857 1181 raft_consensus.cc:383] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.628981 1181 raft_consensus.cc:738] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.629215 1181 consensus_queue.cc:260] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.630427 1181 ts_tablet_manager.cc:1428] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:33.631278 1181 tablet_bootstrap.cc:492] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:33.631371 991 leader_election.cc:304] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:33.631804 991 leader_election.cc:304] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:33.632119 1877 raft_consensus.cc:2798] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:33.632292 1877 raft_consensus.cc:695] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:33.632578 1877 consensus_queue.cc:237] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.633488 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_0dec5de812ab4ea48f1b7a8f4cb0cc4f [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 2 got 0
I20250112 02:52:33.635802 1181 tablet_bootstrap.cc:654] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.635165 2049 raft_consensus.cc:513] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:33.637662 991 leader_election.cc:343] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: daef3a58605e40978f7389698b22265d
I20250112 02:52:33.637712 1177 tablet_bootstrap.cc:492] T 88d69632845547f588662dd56c8c194e P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:33.637861 1177 ts_tablet_manager.cc:1397] T 88d69632845547f588662dd56c8c194e P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.066s user 0.016s sys 0.012s
I20250112 02:52:33.638309 452 catalog_manager.cc:5526] T 039a4734358646508093200d59d30013 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } }
I20250112 02:52:33.638541 1177 raft_consensus.cc:357] T 88d69632845547f588662dd56c8c194e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.638820 1177 raft_consensus.cc:383] T 88d69632845547f588662dd56c8c194e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.638939 1177 raft_consensus.cc:738] T 88d69632845547f588662dd56c8c194e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.639034 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet f924459d1150472aafb325a11825db23 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]). Expected version 3 got 0
I20250112 02:52:33.639227 1177 consensus_queue.cc:260] T 88d69632845547f588662dd56c8c194e P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.640112 2049 leader_election.cc:290] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.640561 1177 ts_tablet_manager.cc:1428] T 88d69632845547f588662dd56c8c194e P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 02:52:33.642516 1177 tablet_bootstrap.cc:492] T bb217afc08304c0ea35dee3120347ba1 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:33.642838 2052 raft_consensus.cc:491] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.643131 2052 raft_consensus.cc:513] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.643978 2052 leader_election.cc:290] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.644292 1181 tablet_bootstrap.cc:492] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:33.644439 1181 ts_tablet_manager.cc:1397] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20250112 02:52:33.644582 1177 tablet_bootstrap.cc:654] T bb217afc08304c0ea35dee3120347ba1 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.644622 989 leader_election.cc:304] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:33.644969 1866 raft_consensus.cc:2798] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:33.645069 1866 raft_consensus.cc:491] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:33.645149 1866 raft_consensus.cc:3054] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.645135 1181 raft_consensus.cc:357] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.645313 1181 raft_consensus.cc:383] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.645377 1181 raft_consensus.cc:738] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.645596 1181 consensus_queue.cc:260] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.646436 1181 ts_tablet_manager.cc:1428] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:33.646703 1181 tablet_bootstrap.cc:492] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:33.648432 1866 raft_consensus.cc:513] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.648765 1181 tablet_bootstrap.cc:654] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.649160 1866 leader_election.cc:290] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:33.649935 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 83348.1us
I20250112 02:52:33.654569 1718 consensus_queue.cc:1035] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:33.655043 1877 raft_consensus.cc:2798] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:33.655320 1877 raft_consensus.cc:695] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:33.656198 1877 consensus_queue.cc:237] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:33.656559 1181 tablet_bootstrap.cc:492] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:33.656688 1181 ts_tablet_manager.cc:1397] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20250112 02:52:33.657379 1181 raft_consensus.cc:357] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.657598 1181 raft_consensus.cc:383] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.657685 1181 raft_consensus.cc:738] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.657927 1181 consensus_queue.cc:260] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.658977 1181 ts_tablet_manager.cc:1428] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:33.659775 1181 tablet_bootstrap.cc:492] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:33.661510 1181 tablet_bootstrap.cc:654] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.639310 452 catalog_manager.cc:5526] T ecfc7d8ba5e4406ebc61f37e6d500473 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } }
I20250112 02:52:33.664021 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb76746b3a846c3aba531c04bf63966" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.665325 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet fccff64af8ca4400afc116db98918119 (table default.Table_15bcb2c0f2a74458a1bed231fd3437fc [id=ca9c8281bf254f248076cb600e7c4f87]). Expected version 2 got 0
I20250112 02:52:33.665593 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_9487626cbda14256af1f17d2675b5d33 [id=485dd281891f45e8b407b306572d6e29]). Expected version 2 got 0
I20250112 02:52:33.665747 2050 consensus_queue.cc:1035] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:33.666213 2032 raft_consensus.cc:491] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.666254 2039 raft_consensus.cc:491] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.666383 2032 raft_consensus.cc:513] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.666386 2039 raft_consensus.cc:513] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.667039 2032 leader_election.cc:290] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:33.667191 2039 leader_election.cc:290] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:33.668159 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61593969d2234ce3a6089027b067d922" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:33.668181 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4abf59d96414e5faa7cf75ba133832c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.668648 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d525dae23744a4adfc2f58cb9945ce" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.669287 2047 raft_consensus.cc:491] T 88d69632845547f588662dd56c8c194e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 02:52:33.669407 855 leader_election.cc:343] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.669471 2047 raft_consensus.cc:513] T 88d69632845547f588662dd56c8c194e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:33.669704 855 leader_election.cc:343] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.670599 1181 tablet_bootstrap.cc:492] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:33.670663 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d69632845547f588662dd56c8c194e" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.670734 1181 ts_tablet_manager.cc:1397] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20250112 02:52:33.670886 1047 raft_consensus.cc:2388] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 289afc116aa74408aa5bff74e11c50d8 in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
I20250112 02:52:33.671422 1181 raft_consensus.cc:357] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:33.671725 1181 raft_consensus.cc:383] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.671837 1181 raft_consensus.cc:738] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.671869 2047 leader_election.cc:290] T 88d69632845547f588662dd56c8c194e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.672137 1181 consensus_queue.cc:260] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:33.672710 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b599e983ec04f53a440d1ab828d12ab" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:33.673122 1181 ts_tablet_manager.cc:1428] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:33.673164 919 raft_consensus.cc:3054] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.673774 1181 tablet_bootstrap.cc:492] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:33.675793 1181 tablet_bootstrap.cc:654] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.677080 919 raft_consensus.cc:2463] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:33.677207 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c69587c5a09d40f0baba5c2f6e8a7b41 (table default.Table_1216acc1fa6b4e51a55a89ffe22bcc9a [id=b5457e4343ee43618c8309e737b69ec3]). Expected version 1 got 0
I20250112 02:52:33.681577 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
I20250112 02:52:33.684376 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb217afc08304c0ea35dee3120347ba1" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.684870 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d69632845547f588662dd56c8c194e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
W20250112 02:52:33.684935 991 leader_election.cc:343] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.685078 771 raft_consensus.cc:3054] T 88d69632845547f588662dd56c8c194e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.685139 2056 raft_consensus.cc:491] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.685266 2056 raft_consensus.cc:513] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.686553 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_f7421d0779924239a3b493954ef748b7 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 1 got 0
I20250112 02:52:33.687121 452 catalog_manager.cc:5526] T e3ee5d8a17874d7ba54de51568e7d2e6 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } }
I20250112 02:52:33.687362 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a18132549374218893f35a4e4226b72" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:33.687566 919 raft_consensus.cc:3054] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.687549 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet e3ee5d8a17874d7ba54de51568e7d2e6 (table default.Table_83177e7dc14645dbb0e5ef573e80e72f [id=2c077da077834ae4bcc84efdd9e89120]). Expected version 1 got 0
I20250112 02:52:33.687873 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d659c5a4f7a446c6aaf2283f65b45106" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.687857 452 catalog_manager.cc:5526] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } }
I20250112 02:52:33.688081 778 raft_consensus.cc:2463] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:33.688483 991 leader_election.cc:304] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:33.688818 1877 raft_consensus.cc:2798] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:33.688966 1877 raft_consensus.cc:491] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:33.689103 1877 raft_consensus.cc:3054] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.689651 771 raft_consensus.cc:2463] T 88d69632845547f588662dd56c8c194e P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:33.690191 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d69632845547f588662dd56c8c194e" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:33.690518 915 raft_consensus.cc:2463] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:33.691603 991 leader_election.cc:304] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:33.691767 1047 raft_consensus.cc:3054] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.692029 2056 leader_election.cc:290] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.692333 2056 raft_consensus.cc:2798] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:33.692508 2056 raft_consensus.cc:695] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:33.692823 2056 consensus_queue.cc:237] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.695627 1734 consensus_queue.cc:1035] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:33.706139 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5cc414d6420c41839621ab858067548b (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]). Expected version 1 got 0
I20250112 02:52:33.706467 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:33.706718 452 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:33.706913 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daef3a58605e40978f7389698b22265d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:33.707144 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb217afc08304c0ea35dee3120347ba1" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:33.707208 915 raft_consensus.cc:2463] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:33.707336 916 raft_consensus.cc:2463] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:33.709270 1877 raft_consensus.cc:513] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.710225 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d659c5a4f7a446c6aaf2283f65b45106" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:33.710458 771 raft_consensus.cc:3054] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.711506 2039 raft_consensus.cc:491] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.711659 2039 raft_consensus.cc:513] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.712219 2039 leader_election.cc:290] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:33.713382 1877 leader_election.cc:290] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.715309 771 raft_consensus.cc:2463] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:33.715641 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b64e4e64cc462891f1ee7a594d5026" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:33.720331 855 leader_election.cc:343] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.720937 991 leader_election.cc:304] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:33.721343 1877 raft_consensus.cc:2798] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:33.721526 1877 raft_consensus.cc:695] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:33.721932 1877 consensus_queue.cc:237] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.723994 1177 tablet_bootstrap.cc:492] T bb217afc08304c0ea35dee3120347ba1 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:33.724217 1177 ts_tablet_manager.cc:1397] T bb217afc08304c0ea35dee3120347ba1 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.082s user 0.008s sys 0.014s
I20250112 02:52:33.725234 1177 raft_consensus.cc:357] T bb217afc08304c0ea35dee3120347ba1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.725503 1177 raft_consensus.cc:383] T bb217afc08304c0ea35dee3120347ba1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.725638 1177 raft_consensus.cc:738] T bb217afc08304c0ea35dee3120347ba1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.725944 1177 consensus_queue.cc:260] T bb217afc08304c0ea35dee3120347ba1 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.729923 1177 ts_tablet_manager.cc:1428] T bb217afc08304c0ea35dee3120347ba1 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250112 02:52:33.730618 1177 tablet_bootstrap.cc:492] T 48634d6d78784fae98512b242e5476df P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:33.731154 713 leader_election.cc:304] T 88d69632845547f588662dd56c8c194e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:33.731787 1718 raft_consensus.cc:2758] T 88d69632845547f588662dd56c8c194e P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250112 02:52:33.737313 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 7e1954e7859f422f9690fe4a9384a8bb (table default.Table_829049fe22b74e62a87dd1bed65fb3fe [id=2e892eba68234ffca0be4ef3e49cf309]). Expected version 1 got 0
I20250112 02:52:33.739413 1177 tablet_bootstrap.cc:654] T 48634d6d78784fae98512b242e5476df P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.742561 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_f7421d0779924239a3b493954ef748b7 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 1 got 0
I20250112 02:52:33.743151 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:33.743908 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
I20250112 02:52:33.744258 1877 raft_consensus.cc:491] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.744556 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5cc414d6420c41839621ab858067548b (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]). Expected version 1 got 0
I20250112 02:52:33.744455 1877 raft_consensus.cc:513] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.745098 454 catalog_manager.cc:5526] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:33.746351 1877 leader_election.cc:290] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.746345 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 7ea075826b16475ca474e284fc6bc545 (table default.Table_009a6976f6db4f218e3c3373368a58d7 [id=c5005669bb5f4c029d673bd39cc1535d]). Expected version 1 got 0
I20250112 02:52:33.746752 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet e3ee5d8a17874d7ba54de51568e7d2e6 (table default.Table_83177e7dc14645dbb0e5ef573e80e72f [id=2c077da077834ae4bcc84efdd9e89120]). Expected version 1 got 0
W20250112 02:52:33.746755 854 leader_election.cc:343] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.746846 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d69632845547f588662dd56c8c194e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
W20250112 02:52:33.746927 991 leader_election.cc:343] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: a1a0a13696fc41ccb28218f1275a4b5d
I20250112 02:52:33.747112 854 leader_election.cc:304] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:33.747166 915 raft_consensus.cc:3054] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.747530 2039 raft_consensus.cc:2743] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:33.747774 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_9487626cbda14256af1f17d2675b5d33 [id=485dd281891f45e8b407b306572d6e29]). Expected version 2 got 0
W20250112 02:52:33.747977 854 leader_election.cc:343] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.748147 854 leader_election.cc:304] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:33.748217 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet fccff64af8ca4400afc116db98918119 (table default.Table_15bcb2c0f2a74458a1bed231fd3437fc [id=ca9c8281bf254f248076cb600e7c4f87]). Expected version 2 got 0
I20250112 02:52:33.748443 2039 raft_consensus.cc:2743] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:33.748956 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet f924459d1150472aafb325a11825db23 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]). Expected version 3 got 0
W20250112 02:52:33.748939 854 leader_election.cc:343] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.749146 854 leader_election.cc:304] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:33.749315 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_feaebd2f4cbb47b6b24ff79614994d6f [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 1 got 0
I20250112 02:52:33.749482 2039 raft_consensus.cc:2743] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:33.749876 854 leader_election.cc:343] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.750026 854 leader_election.cc:304] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:33.750393 2039 raft_consensus.cc:2743] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:33.750666 915 raft_consensus.cc:2463] T 88d69632845547f588662dd56c8c194e P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:33.750386 454 catalog_manager.cc:5526] T 9f0314aa03b1473ebbe05ce73adf7838 P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:33.751780 1177 tablet_bootstrap.cc:492] T 48634d6d78784fae98512b242e5476df P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:33.751842 1173 tablet_bootstrap.cc:492] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:33.752064 1173 ts_tablet_manager.cc:1397] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.139s user 0.005s sys 0.000s
I20250112 02:52:33.752612 1177 ts_tablet_manager.cc:1397] T 48634d6d78784fae98512b242e5476df P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.003s
I20250112 02:52:33.752950 1173 raft_consensus.cc:357] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.753196 1173 raft_consensus.cc:383] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.753302 1173 raft_consensus.cc:738] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.753659 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_0dec5de812ab4ea48f1b7a8f4cb0cc4f [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 2 got 0
I20250112 02:52:33.753593 1173 consensus_queue.cc:260] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.754194 1173 ts_tablet_manager.cc:1428] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:33.754557 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet abfaad67d5e34c0a8338dccc0ef05466 (table default.Table_045e5b104dc64217aa39c22534007cab [id=2c0e94e8c5774da1b8130c7651d94500]). Expected version 1 got 0
I20250112 02:52:33.755005 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 09d06e2e8c13464db8cf801a292bdc99 (table default.Table_0bdc5f49d4ff4d6fb1290bc2dc0f70d9 [id=e3bb489c53454222a79702b798c0c2f1]). Expected version 2 got 0
I20250112 02:52:33.755626 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 82b351c563d749df806ab5d0331a4ef2 (table default.Table_4603068c01544e2e965dca5177e36b40 [id=8502f21e9b61436093db1241cfbb5ec9]). Expected version 1 got 0
I20250112 02:52:33.757304 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:33.757939 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c69587c5a09d40f0baba5c2f6e8a7b41 (table default.Table_1216acc1fa6b4e51a55a89ffe22bcc9a [id=b5457e4343ee43618c8309e737b69ec3]). Expected version 1 got 0
I20250112 02:52:33.758435 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:33.759611 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:33.760277 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:33.760820 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet d8eb0efc5d7f4c72bdfc7531a444d42a (table default.Table_d561e1740cc545dc8aa4babb76b45c37 [id=de27aeff6b3a49b3956498ea5015ff33]). Expected version 1 got 0
I20250112 02:52:33.761636 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c04b64bc2f8348a086ed3c4e59f09b16 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:33.762156 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 4916ae3917084cfe8c93240a8857be0e (table default.Table_69f363c4da234792886b544823b3214e [id=c7052afaf6284ef0972835c1f46cb829]). Expected version 1 got 0
I20250112 02:52:33.762542 1177 raft_consensus.cc:357] T 48634d6d78784fae98512b242e5476df P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.762704 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:33.762830 1177 raft_consensus.cc:383] T 48634d6d78784fae98512b242e5476df P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.762912 1177 raft_consensus.cc:738] T 48634d6d78784fae98512b242e5476df P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.763090 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 23e7a2e88c374e949ced92c557225cbf (table default.Table_cd1f185d82b3474083246ec3702851fc [id=bfa9a504a8d34c86929efb0d15d049d0]). Expected version 3 got 0
I20250112 02:52:33.763253 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ab2e31a65294407bb62b13e92898a7d2 (table default.Table_3321055e22c3410eb5e5fe8be62e004c [id=f8cd4289f89a4a748c26a7a01236406e]). Expected version 2 got 0
I20250112 02:52:33.763177 1177 consensus_queue.cc:260] T 48634d6d78784fae98512b242e5476df P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.763577 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 2c03b5e612d64314b210c11686fdfa2a (table default.Table_4934f6022f2441eda88751236a5798be [id=5c4769bfcefd4fe0a0f3c2e315d50c32]). Expected version 6 got 0
I20250112 02:52:33.763859 1177 ts_tablet_manager.cc:1428] T 48634d6d78784fae98512b242e5476df P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20250112 02:52:33.763921 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:33.764192 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 79c6bdd89f5a44efb56d65ccb724308b (table default.Table_ad69cfacaf7b4050ab2d3d6ed237e7d0 [id=9e72fad141db43c09c7e4692769b6eba]). Expected version 1 got 0
I20250112 02:52:33.764688 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet f8c01b80fc8242d98fdfbf767212144d (table default.Table_63649e9607394f4bb72a3dd6e6d0dd69 [id=9807191b6e5c4d478cd47a54464fb9ce]). Expected version 1 got 0
I20250112 02:52:33.764918 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 6117dc8b32a64535a2f4167a03bed0d2 (table default.Table_365739770a11434dbf260ea912e7ea81 [id=868adb627654450c936ef198e49d853b]). Expected version 1 got 0
I20250112 02:52:33.765197 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:33.765467 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 8ee98f055024429eafaa29bb961cab87 (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 1 got 0
I20250112 02:52:33.765601 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet f31b777e7e0f43a28839d8c48c6a6590 (table default.Table_5fb592e41cc4406aa27546e7b41fdb88 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 1 got 0
I20250112 02:52:33.765826 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:33.765847 1177 tablet_bootstrap.cc:492] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:33.765964 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 9af55dab202444358ea1a0674d37946d (table default.Table_26dfe346379948b39f90b31653c2a463 [id=6c6f30e60e79451397841d090d11d8b7]). Expected version 1 got 0
I20250112 02:52:33.766222 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:33.766556 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 003a3b4705e943c6a900762ec11ddd47 (table default.Table_743b3f7029534564b35c0c137b35fd75 [id=b2fa3922f6d447769a8179052f9a1861]). Expected version 1 got 0
I20250112 02:52:33.767078 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 7cd9985cd20148f195a857c478b0755f (table default.Table_9b65dd291c9f4fcdaa6d5e37f04d24ae [id=e63daf724d874c89badafb8d59653f74]). Expected version 1 got 0
I20250112 02:52:33.768922 1177 tablet_bootstrap.cc:654] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.772024 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:33.772567 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:33.773108 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
W20250112 02:52:33.774587 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca6f080e5fd84084b601b646d4e6c9e2 consensus state has no leader
W20250112 02:52:33.750753 854 leader_election.cc:343] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.774827 854 leader_election.cc:304] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:33.775099 2039 raft_consensus.cc:2743] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:33.775074 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_31424b670f274fd3abd17eb8451ba95c" } modify_external_catalogs: true
W20250112 02:52:33.775278 854 leader_election.cc:343] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.775259 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_31424b670f274fd3abd17eb8451ba95c" } modify_external_catalogs: true
I20250112 02:52:33.775373 854 leader_election.cc:304] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:33.762115 2071 tablet.cc:1721] T 79c6bdd89f5a44efb56d65ccb724308b P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:33.776643 2039 raft_consensus.cc:2743] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:33.782943 1173 tablet_bootstrap.cc:492] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:33.783439 2052 raft_consensus.cc:491] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.783653 2052 raft_consensus.cc:513] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.784373 2052 leader_election.cc:290] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.785094 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48634d6d78784fae98512b242e5476df" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.785341 771 raft_consensus.cc:2463] T 48634d6d78784fae98512b242e5476df P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:33.785776 991 leader_election.cc:304] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:33.786185 2052 raft_consensus.cc:2798] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:33.786296 2052 raft_consensus.cc:491] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:33.786417 2052 raft_consensus.cc:3054] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:33.787282 854 leader_election.cc:343] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.787433 854 leader_election.cc:304] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:33.789953 2039 raft_consensus.cc:2743] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:33.790491 854 leader_election.cc:343] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.790627 854 leader_election.cc:304] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:33.791344 1173 tablet_bootstrap.cc:654] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.791642 2052 raft_consensus.cc:513] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.791857 2039 raft_consensus.cc:2743] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:33.792196 1181 tablet_bootstrap.cc:492] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:33.792337 1181 ts_tablet_manager.cc:1397] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.119s user 0.029s sys 0.035s
I20250112 02:52:33.792989 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48634d6d78784fae98512b242e5476df" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:33.793088 2052 leader_election.cc:290] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.793267 771 raft_consensus.cc:3054] T 48634d6d78784fae98512b242e5476df P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.797004 771 raft_consensus.cc:2463] T 48634d6d78784fae98512b242e5476df P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:33.797583 991 leader_election.cc:304] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
W20250112 02:52:33.797761 854 leader_election.cc:343] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.797688 1181 raft_consensus.cc:357] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.797857 854 leader_election.cc:304] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:33.798033 2052 raft_consensus.cc:2798] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:33.798592 2052 raft_consensus.cc:695] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:33.797907 1181 raft_consensus.cc:383] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.798960 1181 raft_consensus.cc:738] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.799047 2039 raft_consensus.cc:2743] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:33.798975 2052 consensus_queue.cc:237] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.799208 1181 consensus_queue.cc:260] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.799825 1181 ts_tablet_manager.cc:1428] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250112 02:52:33.800138 1181 tablet_bootstrap.cc:492] T e2f1f1a76e5f4308bf56dd647f02c6d8 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:33.774521 919 raft_consensus.cc:2463] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:33.804755 1177 tablet_bootstrap.cc:492] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:33.804953 1177 ts_tablet_manager.cc:1397] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.000s
I20250112 02:52:33.805761 1177 raft_consensus.cc:357] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.806839 1177 raft_consensus.cc:383] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.807098 1177 raft_consensus.cc:738] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.808493 1177 consensus_queue.cc:260] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.812325 1177 ts_tablet_manager.cc:1428] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250112 02:52:33.813676 1177 tablet_bootstrap.cc:492] T 9d5b3df88a0c45dba879b3d506710614 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:33.814106 1181 tablet_bootstrap.cc:654] T e2f1f1a76e5f4308bf56dd647f02c6d8 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.816295 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_f7421d0779924239a3b493954ef748b7 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 1 got 0
I20250112 02:52:33.816491 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:33.816830 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:33.816984 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:33.818537 1177 tablet_bootstrap.cc:654] T 9d5b3df88a0c45dba879b3d506710614 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.821137 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4abf59d96414e5faa7cf75ba133832c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.821976 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d525dae23744a4adfc2f58cb9945ce" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.822500 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_9487626cbda14256af1f17d2675b5d33 [id=485dd281891f45e8b407b306572d6e29]). Expected version 2 got 0
I20250112 02:52:33.822511 989 leader_election.cc:304] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:33.822911 2052 raft_consensus.cc:2743] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:33.823124 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet e3ee5d8a17874d7ba54de51568e7d2e6 (table default.Table_83177e7dc14645dbb0e5ef573e80e72f [id=2c077da077834ae4bcc84efdd9e89120]). Expected version 1 got 0
I20250112 02:52:33.823596 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet fccff64af8ca4400afc116db98918119 (table default.Table_15bcb2c0f2a74458a1bed231fd3437fc [id=ca9c8281bf254f248076cb600e7c4f87]). Expected version 2 got 0
I20250112 02:52:33.823798 989 leader_election.cc:304] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:33.824262 2052 raft_consensus.cc:2798] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:33.824402 2052 raft_consensus.cc:491] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:33.824524 2052 raft_consensus.cc:3054] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.825909 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet f924459d1150472aafb325a11825db23 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]). Expected version 3 got 0
I20250112 02:52:33.826354 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
I20250112 02:52:33.826797 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet abfaad67d5e34c0a8338dccc0ef05466 (table default.Table_045e5b104dc64217aa39c22534007cab [id=2c0e94e8c5774da1b8130c7651d94500]). Expected version 1 got 0
I20250112 02:52:33.827044 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 09d06e2e8c13464db8cf801a292bdc99 (table default.Table_0bdc5f49d4ff4d6fb1290bc2dc0f70d9 [id=e3bb489c53454222a79702b798c0c2f1]). Expected version 2 got 0
I20250112 02:52:33.827785 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet ab2e31a65294407bb62b13e92898a7d2 (table default.Table_3321055e22c3410eb5e5fe8be62e004c [id=f8cd4289f89a4a748c26a7a01236406e]). Expected version 2 got 0
I20250112 02:52:33.827862 2052 raft_consensus.cc:513] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.828472 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet d8eb0efc5d7f4c72bdfc7531a444d42a (table default.Table_d561e1740cc545dc8aa4babb76b45c37 [id=de27aeff6b3a49b3956498ea5015ff33]). Expected version 1 got 0
W20250112 02:52:33.828938 991 leader_election.cc:343] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: daef3a58605e40978f7389698b22265d
I20250112 02:52:33.829250 2052 leader_election.cc:290] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.829594 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 2c03b5e612d64314b210c11686fdfa2a (table default.Table_4934f6022f2441eda88751236a5798be [id=5c4769bfcefd4fe0a0f3c2e315d50c32]). Expected version 6 got 0
I20250112 02:52:33.829649 989 leader_election.cc:304] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:33.829989 2052 raft_consensus.cc:2798] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:33.830072 2052 raft_consensus.cc:491] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:33.830456 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_0dec5de812ab4ea48f1b7a8f4cb0cc4f [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 2 got 0
I20250112 02:52:33.830699 2052 raft_consensus.cc:3054] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.831807 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 4916ae3917084cfe8c93240a8857be0e (table default.Table_69f363c4da234792886b544823b3214e [id=c7052afaf6284ef0972835c1f46cb829]). Expected version 1 got 0
I20250112 02:52:33.832839 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 82b351c563d749df806ab5d0331a4ef2 (table default.Table_4603068c01544e2e965dca5177e36b40 [id=8502f21e9b61436093db1241cfbb5ec9]). Expected version 1 got 0
I20250112 02:52:33.833077 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b64e4e64cc462891f1ee7a594d5026" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:33.838007 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 23e7a2e88c374e949ced92c557225cbf (table default.Table_cd1f185d82b3474083246ec3702851fc [id=bfa9a504a8d34c86929efb0d15d049d0]). Expected version 3 got 0
I20250112 02:52:33.838197 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:33.838449 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c04b64bc2f8348a086ed3c4e59f09b16 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:33.838677 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 7e1954e7859f422f9690fe4a9384a8bb (table default.Table_829049fe22b74e62a87dd1bed65fb3fe [id=2e892eba68234ffca0be4ef3e49cf309]). Expected version 1 got 0
I20250112 02:52:33.839033 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 5cc414d6420c41839621ab858067548b (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]). Expected version 1 got 0
I20250112 02:52:33.839877 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:33.840121 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d659c5a4f7a446c6aaf2283f65b45106" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:33.840255 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:33.840462 918 raft_consensus.cc:2463] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:33.840759 453 catalog_manager.cc:5526] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:33.841538 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 7ea075826b16475ca474e284fc6bc545 (table default.Table_009a6976f6db4f218e3c3373368a58d7 [id=c5005669bb5f4c029d673bd39cc1535d]). Expected version 1 got 0
I20250112 02:52:33.841753 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d659c5a4f7a446c6aaf2283f65b45106" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:33.841893 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 79c6bdd89f5a44efb56d65ccb724308b (table default.Table_ad69cfacaf7b4050ab2d3d6ed237e7d0 [id=9e72fad141db43c09c7e4692769b6eba]). Expected version 1 got 0
I20250112 02:52:33.842083 918 raft_consensus.cc:3054] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.842410 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet f8c01b80fc8242d98fdfbf767212144d (table default.Table_63649e9607394f4bb72a3dd6e6d0dd69 [id=9807191b6e5c4d478cd47a54464fb9ce]). Expected version 1 got 0
I20250112 02:52:33.842609 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 6117dc8b32a64535a2f4167a03bed0d2 (table default.Table_365739770a11434dbf260ea912e7ea81 [id=868adb627654450c936ef198e49d853b]). Expected version 1 got 0
I20250112 02:52:33.842952 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:33.843116 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 8ee98f055024429eafaa29bb961cab87 (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 1 got 0
I20250112 02:52:33.843286 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet f31b777e7e0f43a28839d8c48c6a6590 (table default.Table_5fb592e41cc4406aa27546e7b41fdb88 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 1 got 0
I20250112 02:52:33.843514 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c69587c5a09d40f0baba5c2f6e8a7b41 (table default.Table_1216acc1fa6b4e51a55a89ffe22bcc9a [id=b5457e4343ee43618c8309e737b69ec3]). Expected version 1 got 0
I20250112 02:52:33.843658 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:33.843861 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:33.844131 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:33.844290 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 9af55dab202444358ea1a0674d37946d (table default.Table_26dfe346379948b39f90b31653c2a463 [id=6c6f30e60e79451397841d090d11d8b7]). Expected version 1 got 0
I20250112 02:52:33.844550 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 003a3b4705e943c6a900762ec11ddd47 (table default.Table_743b3f7029534564b35c0c137b35fd75 [id=b2fa3922f6d447769a8179052f9a1861]). Expected version 1 got 0
I20250112 02:52:33.845011 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 7cd9985cd20148f195a857c478b0755f (table default.Table_9b65dd291c9f4fcdaa6d5e37f04d24ae [id=e63daf724d874c89badafb8d59653f74]). Expected version 1 got 0
I20250112 02:52:33.845175 453 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:33.845937 918 raft_consensus.cc:2463] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:33.846182 1177 tablet_bootstrap.cc:492] T 9d5b3df88a0c45dba879b3d506710614 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:33.846340 1177 ts_tablet_manager.cc:1397] T 9d5b3df88a0c45dba879b3d506710614 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.004s
I20250112 02:52:33.847007 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48634d6d78784fae98512b242e5476df" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:33.847206 919 raft_consensus.cc:2463] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:33.847148 1177 raft_consensus.cc:357] T 9d5b3df88a0c45dba879b3d506710614 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.847370 1177 raft_consensus.cc:383] T 9d5b3df88a0c45dba879b3d506710614 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.847448 1177 raft_consensus.cc:738] T 9d5b3df88a0c45dba879b3d506710614 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.847765 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48634d6d78784fae98512b242e5476df" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:33.847719 1177 consensus_queue.cc:260] T 9d5b3df88a0c45dba879b3d506710614 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.847926 919 raft_consensus.cc:3054] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.848268 1177 ts_tablet_manager.cc:1428] T 9d5b3df88a0c45dba879b3d506710614 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 02:52:33.851069 919 raft_consensus.cc:2463] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
W20250112 02:52:33.851739 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e67f0a0c065498f9a644220414415b0 consensus state has no leader
I20250112 02:52:33.852149 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_829049fe22b74e62a87dd1bed65fb3fe" } modify_external_catalogs: true
I20250112 02:52:33.852335 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_829049fe22b74e62a87dd1bed65fb3fe" } modify_external_catalogs: true
I20250112 02:52:33.854588 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daef3a58605e40978f7389698b22265d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:33.854813 917 raft_consensus.cc:3054] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.855291 1177 tablet_bootstrap.cc:492] T fccff64af8ca4400afc116db98918119 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:33.857110 1177 tablet_bootstrap.cc:654] T fccff64af8ca4400afc116db98918119 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.858423 2039 raft_consensus.cc:491] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.858582 2039 raft_consensus.cc:513] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.859196 2039 leader_election.cc:290] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:33.859724 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c10fad790046a88fbc8ebe0303c481" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:33.859833 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bb4ef0220dde43569452379639759db5 consensus state has no leader
W20250112 02:52:33.860194 855 leader_election.cc:343] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.860216 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_26dfe346379948b39f90b31653c2a463" } modify_external_catalogs: true
I20250112 02:52:33.860394 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_26dfe346379948b39f90b31653c2a463" } modify_external_catalogs: true
I20250112 02:52:33.859136 917 raft_consensus.cc:2463] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:33.862284 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet bd5b3c712c0f448ea9b14ab93dd731ab
I20250112 02:52:33.863173 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_c816af31a1e945518fd886cbf8b8b885"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:33.863803 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c816af31a1e945518fd886cbf8b8b885 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:33.864188 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 1dda52f012794493a102ed1abc718f5b from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:33.865032 892 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5b3c712c0f448ea9b14ab93dd731ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:33.865075 760 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5b3c712c0f448ea9b14ab93dd731ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:33.865841 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet bd5b3c712c0f448ea9b14ab93dd731ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd5b3c712c0f448ea9b14ab93dd731ab already in progress: creating tablet
W20250112 02:52:33.866072 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet bd5b3c712c0f448ea9b14ab93dd731ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd5b3c712c0f448ea9b14ab93dd731ab already in progress: creating tablet
I20250112 02:52:33.866362 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet d63d411f906b4d4eb68343bdcc493e05
I20250112 02:52:33.867161 892 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d411f906b4d4eb68343bdcc493e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:33.867079 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_23bc94e376f645a79c55f7f22bb15e77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:33.867583 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23bc94e376f645a79c55f7f22bb15e77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:33.867820 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet d63d411f906b4d4eb68343bdcc493e05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d63d411f906b4d4eb68343bdcc493e05 already in progress: creating tablet
I20250112 02:52:33.869266 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5b3c712c0f448ea9b14ab93dd731ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:33.869570 760 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d411f906b4d4eb68343bdcc493e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:33.869721 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41d5799b5b36487da42dc43b1c70ae26 consensus state has no leader
I20250112 02:52:33.870112 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_489753b0c1154a4ba82ad30de84de181"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:33.870548 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_489753b0c1154a4ba82ad30de84de181 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:33.871413 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d411f906b4d4eb68343bdcc493e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:33.871747 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d63d411f906b4d4eb68343bdcc493e05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d63d411f906b4d4eb68343bdcc493e05 already in progress: creating tablet
I20250112 02:52:33.872045 446 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet fbe9e9624ede40c29ea852b2b5494965
I20250112 02:52:33.872272 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 55fb1a197d7e40689ccad82a44492371 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:33.872368 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 54215c55966a46e1ab96e352c4e3764e from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:33.875975 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_7870d93893b74b4b96d1a3809a9015e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:33.876449 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7870d93893b74b4b96d1a3809a9015e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:33.876737 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_ea881fde8d694f71b5faf60948630d96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:33.877161 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea881fde8d694f71b5faf60948630d96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:33.879009 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet bd5b3c712c0f448ea9b14ab93dd731ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd5b3c712c0f448ea9b14ab93dd731ab already in progress: creating tablet
W20250112 02:52:33.879734 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet d63d411f906b4d4eb68343bdcc493e05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d63d411f906b4d4eb68343bdcc493e05 already in progress: creating tablet
I20250112 02:52:33.889451 2050 consensus_queue.cc:1035] T ed7243c6329d4f11aa25eea2b7b67683 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:33.894402 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_b1a770633f004b01b66ddfa83e0fd698"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:33.894954 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1a770633f004b01b66ddfa83e0fd698 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:33.896220 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_9d8f183a5347440f898ef33ee7299613"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:33.896649 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d8f183a5347440f898ef33ee7299613 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:33.904958 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 1dda52f012794493a102ed1abc718f5b
I20250112 02:52:33.905397 447 catalog_manager.cc:6596] ReplaceTablet: tablet 1dda52f012794493a102ed1abc718f5b deleted and replaced by tablet 17c54c5dc86446cf85974777c8145ef6
I20250112 02:52:33.905870 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_8f55d3bee66c40239788d75612403ca6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:33.906273 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f55d3bee66c40239788d75612403ca6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:33.908525 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 54215c55966a46e1ab96e352c4e3764e
I20250112 02:52:33.908643 2052 raft_consensus.cc:513] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.909461 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb217afc08304c0ea35dee3120347ba1" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:33.909641 778 raft_consensus.cc:3054] T bb217afc08304c0ea35dee3120347ba1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:33.909737 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dda52f012794493a102ed1abc718f5b with delete_type TABLET_DATA_DELETED (replaced by tablet 17c54c5dc86446cf85974777c8145ef6) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:33.909981 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 54215c55966a46e1ab96e352c4e3764e with delete_type TABLET_DATA_DELETED (replaced by tablet a470f84d50df4cf0b4a1e5d248b4c32a) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:33.910264 2036 tablet_replica.cc:331] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:33.910540 2036 raft_consensus.cc:2238] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:33.910717 2036 raft_consensus.cc:2267] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:33.911278 2036 ts_tablet_manager.cc:1905] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:33.912025 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dda52f012794493a102ed1abc718f5b with delete_type TABLET_DATA_DELETED (replaced by tablet 17c54c5dc86446cf85974777c8145ef6) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:33.912771 778 raft_consensus.cc:2463] T bb217afc08304c0ea35dee3120347ba1 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:33.912950 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 54215c55966a46e1ab96e352c4e3764e with delete_type TABLET_DATA_DELETED (replaced by tablet a470f84d50df4cf0b4a1e5d248b4c32a) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:33.913338 991 leader_election.cc:304] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:33.913671 2050 raft_consensus.cc:2798] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:33.913801 2050 raft_consensus.cc:695] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:33.914122 2050 consensus_queue.cc:237] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.916527 455 catalog_manager.cc:6596] ReplaceTablet: tablet 54215c55966a46e1ab96e352c4e3764e deleted and replaced by tablet a470f84d50df4cf0b4a1e5d248b4c32a
I20250112 02:52:33.916724 455 master_service.cc:946] ReplaceTablet: received request to replace tablet d63d411f906b4d4eb68343bdcc493e05 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:33.923117 1727 raft_consensus.cc:491] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.923298 1727 raft_consensus.cc:513] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.924077 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 7e1954e7859f422f9690fe4a9384a8bb
W20250112 02:52:33.924633 991 leader_election.cc:343] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: aba1744138fe42ae9d097db01cac083d
I20250112 02:52:33.924892 1727 leader_election.cc:290] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.925429 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dda52f012794493a102ed1abc718f5b with delete_type TABLET_DATA_DELETED (replaced by tablet 17c54c5dc86446cf85974777c8145ef6) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:33.925618 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1954e7859f422f9690fe4a9384a8bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:33.925683 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 54215c55966a46e1ab96e352c4e3764e with delete_type TABLET_DATA_DELETED (replaced by tablet a470f84d50df4cf0b4a1e5d248b4c32a) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:33.926399 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1954e7859f422f9690fe4a9384a8bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:33.927239 2056 consensus_queue.cc:1035] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:33.927917 448 master_service.cc:946] ReplaceTablet: received request to replace tablet c04b64bc2f8348a086ed3c4e59f09b16 from {username='slave'} at 127.0.0.1:40610
W20250112 02:52:33.929167 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 1dda52f012794493a102ed1abc718f5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1dda52f012794493a102ed1abc718f5b already in progress: creating tablet
W20250112 02:52:33.929764 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 54215c55966a46e1ab96e352c4e3764e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54215c55966a46e1ab96e352c4e3764e already in progress: creating tablet
W20250112 02:52:33.930163 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 1dda52f012794493a102ed1abc718f5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1dda52f012794493a102ed1abc718f5b already in progress: creating tablet
W20250112 02:52:33.934247 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 7e1954e7859f422f9690fe4a9384a8bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e1954e7859f422f9690fe4a9384a8bb already in progress: creating tablet
W20250112 02:52:33.934514 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 54215c55966a46e1ab96e352c4e3764e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54215c55966a46e1ab96e352c4e3764e already in progress: creating tablet
I20250112 02:52:33.935380 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 55fb1a197d7e40689ccad82a44492371
I20250112 02:52:33.935914 446 catalog_manager.cc:6596] ReplaceTablet: tablet 55fb1a197d7e40689ccad82a44492371 deleted and replaced by tablet a8e7882e07784bdda18dc1ffd20963c8
I20250112 02:52:33.936873 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 55fb1a197d7e40689ccad82a44492371 with delete_type TABLET_DATA_DELETED (replaced by tablet a8e7882e07784bdda18dc1ffd20963c8) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:33.938707 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 55fb1a197d7e40689ccad82a44492371 with delete_type TABLET_DATA_DELETED (replaced by tablet a8e7882e07784bdda18dc1ffd20963c8) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:33.938956 2003 tablet_replica.cc:331] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:33.939088 2036 ts_tablet_manager.cc:1918] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:33.939162 2003 raft_consensus.cc:2238] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:33.939229 2036 log.cc:1198] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/1dda52f012794493a102ed1abc718f5b
I20250112 02:52:33.939505 2003 raft_consensus.cc:2267] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:33.939805 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1954e7859f422f9690fe4a9384a8bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:33.940070 2003 ts_tablet_manager.cc:1905] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:33.940575 2036 ts_tablet_manager.cc:1939] T 1dda52f012794493a102ed1abc718f5b P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:33.941749 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 1dda52f012794493a102ed1abc718f5b (table default.Table_ab4d7c58924a4b8682b52637c23f125b [id=a622f7a973344a388b3275adbcce6234]) successfully deleted
W20250112 02:52:33.942747 1778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:52:33.943199 1778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:52:33.943358 1778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250112 02:52:33.943816 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_5e1a058646834ea99def3d8262bee81b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:33.944360 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e1a058646834ea99def3d8262bee81b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:33.935288 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_d5f4d4c431d24cc0ba82a7dcd012368b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:33.945858 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5f4d4c431d24cc0ba82a7dcd012368b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:33.947213 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 7e1954e7859f422f9690fe4a9384a8bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e1954e7859f422f9690fe4a9384a8bb already in progress: creating tablet
I20250112 02:52:33.949533 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 9af55dab202444358ea1a0674d37946d
I20250112 02:52:33.950397 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af55dab202444358ea1a0674d37946d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:33.950501 2003 ts_tablet_manager.cc:1918] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:33.950639 2003 log.cc:1198] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/55fb1a197d7e40689ccad82a44492371
I20250112 02:52:33.951081 2003 ts_tablet_manager.cc:1939] T 55fb1a197d7e40689ccad82a44492371 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:33.951360 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af55dab202444358ea1a0674d37946d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:33.952576 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 55fb1a197d7e40689ccad82a44492371 (table default.Table_f3695b00a89b4353903587a6c1418d8c [id=d1781738b8a74f139b09863918cbb21d]) successfully deleted
W20250112 02:52:33.952838 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 9af55dab202444358ea1a0674d37946d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9af55dab202444358ea1a0674d37946d already in progress: creating tablet
I20250112 02:52:33.956913 2052 leader_election.cc:290] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.964171 1181 tablet_bootstrap.cc:492] T e2f1f1a76e5f4308bf56dd647f02c6d8 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:33.964301 1181 ts_tablet_manager.cc:1397] T e2f1f1a76e5f4308bf56dd647f02c6d8 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.164s user 0.028s sys 0.016s
I20250112 02:52:33.964660 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 55fb1a197d7e40689ccad82a44492371 with delete_type TABLET_DATA_DELETED (replaced by tablet a8e7882e07784bdda18dc1ffd20963c8) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:33.964901 2024 tablet_replica.cc:331] T 55fb1a197d7e40689ccad82a44492371 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:33.964996 1181 raft_consensus.cc:357] T e2f1f1a76e5f4308bf56dd647f02c6d8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:33.965123 2024 raft_consensus.cc:2238] T 55fb1a197d7e40689ccad82a44492371 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:33.965157 1181 raft_consensus.cc:383] T e2f1f1a76e5f4308bf56dd647f02c6d8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.965204 1181 raft_consensus.cc:738] T e2f1f1a76e5f4308bf56dd647f02c6d8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.965319 2024 raft_consensus.cc:2267] T 55fb1a197d7e40689ccad82a44492371 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:33.965358 1181 consensus_queue.cc:260] T e2f1f1a76e5f4308bf56dd647f02c6d8 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:33.965866 2024 ts_tablet_manager.cc:1905] T 55fb1a197d7e40689ccad82a44492371 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:33.965921 1181 ts_tablet_manager.cc:1428] T e2f1f1a76e5f4308bf56dd647f02c6d8 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:33.966192 1181 tablet_bootstrap.cc:492] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:33.967454 2052 raft_consensus.cc:491] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.967657 1181 tablet_bootstrap.cc:654] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:33.969805 2071 tablet.cc:1721] T ab2e31a65294407bb62b13e92898a7d2 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:33.972519 1181 tablet_bootstrap.cc:492] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:33.972645 1181 ts_tablet_manager.cc:1397] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20250112 02:52:33.972700 2024 ts_tablet_manager.cc:1918] T 55fb1a197d7e40689ccad82a44492371 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:33.973577 1181 raft_consensus.cc:357] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.973781 1181 raft_consensus.cc:383] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.973855 1181 raft_consensus.cc:738] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.974121 1181 consensus_queue.cc:260] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.974785 1181 ts_tablet_manager.cc:1428] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:33.975072 1181 tablet_bootstrap.cc:492] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:33.975150 2052 raft_consensus.cc:513] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.976840 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccff64af8ca4400afc116db98918119" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.977598 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_952b91d2b0b8435886e555f12629dc80" } new_table_name: "default.Table_c5704699453f40298d94c2f5dbd10179" modify_external_catalogs: true
W20250112 02:52:33.978039 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 54215c55966a46e1ab96e352c4e3764e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54215c55966a46e1ab96e352c4e3764e already in progress: creating tablet
W20250112 02:52:33.978600 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 7e1954e7859f422f9690fe4a9384a8bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e1954e7859f422f9690fe4a9384a8bb already in progress: creating tablet
I20250112 02:52:33.979087 2036 tablet_replica.cc:331] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:33.977108 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af55dab202444358ea1a0674d37946d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:33.977372 991 leader_election.cc:343] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:33.977133 2024 log.cc:1198] T 55fb1a197d7e40689ccad82a44492371 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/55fb1a197d7e40689ccad82a44492371
I20250112 02:52:33.981334 2024 ts_tablet_manager.cc:1939] T 55fb1a197d7e40689ccad82a44492371 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:33.982738 2094 tablet.cc:1721] T ab2e31a65294407bb62b13e92898a7d2 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:33.983129 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 55fb1a197d7e40689ccad82a44492371 (table default.Table_f3695b00a89b4353903587a6c1418d8c [id=d1781738b8a74f139b09863918cbb21d]) successfully deleted
I20250112 02:52:33.983222 1181 tablet_bootstrap.cc:654] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:33.983844 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 9af55dab202444358ea1a0674d37946d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9af55dab202444358ea1a0674d37946d already in progress: creating tablet
I20250112 02:52:33.984581 2036 raft_consensus.cc:2238] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:33.984782 2036 raft_consensus.cc:2267] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:33.985370 2036 ts_tablet_manager.cc:1905] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:33.986078 2039 raft_consensus.cc:491] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:33.986217 2039 raft_consensus.cc:513] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.986863 2039 leader_election.cc:290] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:33.977526 1173 tablet_bootstrap.cc:492] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:33.988675 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet c04b64bc2f8348a086ed3c4e59f09b16
I20250112 02:52:33.988791 1173 ts_tablet_manager.cc:1397] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.213s user 0.007s sys 0.001s
I20250112 02:52:33.989207 448 catalog_manager.cc:6596] ReplaceTablet: tablet c04b64bc2f8348a086ed3c4e59f09b16 deleted and replaced by tablet 61eebc5f22664712af6e74e82809ca87
I20250112 02:52:33.989658 898 tablet_service.cc:1514] Processing DeleteTablet for tablet c04b64bc2f8348a086ed3c4e59f09b16 with delete_type TABLET_DATA_DELETED (replaced by tablet 61eebc5f22664712af6e74e82809ca87) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:33.989655 1173 raft_consensus.cc:357] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.990352 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet c04b64bc2f8348a086ed3c4e59f09b16 with delete_type TABLET_DATA_DELETED (replaced by tablet 61eebc5f22664712af6e74e82809ca87) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:33.992666 2024 tablet_replica.cc:331] T c04b64bc2f8348a086ed3c4e59f09b16 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:33.992982 2024 raft_consensus.cc:2238] T c04b64bc2f8348a086ed3c4e59f09b16 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:33.993254 2024 raft_consensus.cc:2267] T c04b64bc2f8348a086ed3c4e59f09b16 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:33.993906 2024 ts_tablet_manager.cc:1905] T c04b64bc2f8348a086ed3c4e59f09b16 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:33.976776 2052 leader_election.cc:290] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:33.995523 1173 raft_consensus.cc:383] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:33.995641 1173 raft_consensus.cc:738] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:33.995844 1173 consensus_queue.cc:260] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:33.996506 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_67e40ef8ee824430b5d090ddc9cc03c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:33.997002 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67e40ef8ee824430b5d090ddc9cc03c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:33.987411 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f234256b343f447dadfc3d1936a824ee" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:33.990141 744 tablet_service.cc:1514] Processing DeleteTablet for tablet c04b64bc2f8348a086ed3c4e59f09b16 with delete_type TABLET_DATA_DELETED (replaced by tablet 61eebc5f22664712af6e74e82809ca87) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:33.997831 1173 ts_tablet_manager.cc:1428] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250112 02:52:33.997879 2003 tablet_replica.cc:331] T c04b64bc2f8348a086ed3c4e59f09b16 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:33.998085 2003 raft_consensus.cc:2238] T c04b64bc2f8348a086ed3c4e59f09b16 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:33.998162 2028 tablet.cc:1721] T ab2e31a65294407bb62b13e92898a7d2 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:33.998245 2003 raft_consensus.cc:2267] T c04b64bc2f8348a086ed3c4e59f09b16 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:33.998751 2003 ts_tablet_manager.cc:1905] T c04b64bc2f8348a086ed3c4e59f09b16 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:34.001415 855 leader_election.cc:343] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.007109 2003 ts_tablet_manager.cc:1918] T c04b64bc2f8348a086ed3c4e59f09b16 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:34.007277 2003 log.cc:1198] T c04b64bc2f8348a086ed3c4e59f09b16 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/c04b64bc2f8348a086ed3c4e59f09b16
I20250112 02:52:34.007751 2003 ts_tablet_manager.cc:1939] T c04b64bc2f8348a086ed3c4e59f09b16 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:34.008802 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet c04b64bc2f8348a086ed3c4e59f09b16 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]) successfully deleted
I20250112 02:52:34.009931 1734 consensus_queue.cc:1035] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:34.013103 1181 tablet_bootstrap.cc:492] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:34.013217 1181 ts_tablet_manager.cc:1397] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.003s
I20250112 02:52:34.013527 1177 tablet_bootstrap.cc:492] T fccff64af8ca4400afc116db98918119 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:34.013685 1177 ts_tablet_manager.cc:1397] T fccff64af8ca4400afc116db98918119 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.158s user 0.004s sys 0.005s
I20250112 02:52:34.013837 1181 raft_consensus.cc:357] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.014000 1181 raft_consensus.cc:383] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:34.014045 1181 raft_consensus.cc:738] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:34.014217 1181 consensus_queue.cc:260] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.014675 1181 ts_tablet_manager.cc:1428] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:34.014928 1181 tablet_bootstrap.cc:492] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:34.016443 1718 consensus_queue.cc:1035] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:34.016584 1181 tablet_bootstrap.cc:654] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:34.018556 1181 tablet_bootstrap.cc:492] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:34.018651 1181 ts_tablet_manager.cc:1397] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250112 02:52:34.019207 1181 raft_consensus.cc:357] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.019631 1181 raft_consensus.cc:383] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:34.019690 1181 raft_consensus.cc:738] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:34.019850 1181 consensus_queue.cc:260] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.020268 1181 ts_tablet_manager.cc:1428] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:34.020503 1181 tablet_bootstrap.cc:492] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:34.020740 1177 raft_consensus.cc:357] T fccff64af8ca4400afc116db98918119 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.022238 1181 tablet_bootstrap.cc:654] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:34.022694 1727 raft_consensus.cc:491] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.022898 1727 raft_consensus.cc:513] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.023523 1727 leader_election.cc:290] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:34.024417 1173 tablet_bootstrap.cc:492] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:34.026243 1173 tablet_bootstrap.cc:654] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:34.027415 2036 ts_tablet_manager.cc:1918] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:34.023890 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b70dce4d1d4e29997a18297ac0f82a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:34.029978 1177 raft_consensus.cc:383] T fccff64af8ca4400afc116db98918119 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:34.030086 1177 raft_consensus.cc:738] T fccff64af8ca4400afc116db98918119 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:34.030325 1177 consensus_queue.cc:260] T fccff64af8ca4400afc116db98918119 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.031036 1177 ts_tablet_manager.cc:1428] T fccff64af8ca4400afc116db98918119 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250112 02:52:34.032497 1177 tablet_bootstrap.cc:492] T 1a1e3af2883341e9b957f6ed748e604d P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:34.034025 1177 tablet_bootstrap.cc:654] T 1a1e3af2883341e9b957f6ed748e604d P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:34.040086 826 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
W20250112 02:52:34.040807 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 41d5799b5b36487da42dc43b1c70ae26 consensus state has no leader
W20250112 02:52:34.048703 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5079f87652a4ea58b366d1f9448fd4e has no consensus state
I20250112 02:52:34.054934 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_23bc94e376f645a79c55f7f22bb15e77" } new_table_name: "default.Table_efed5d5c2a7b421eba5e50bf60c00bce" modify_external_catalogs: true
I20250112 02:52:34.056422 760 tablet_service.cc:1467] Processing CreateTablet for tablet ff1b3ee229c445b19405cf4a19cf8003 (DEFAULT_TABLE table=default.Table_6b8cb70a5e5b4519af5d247f6b5bb8b0 [id=a28f5cfbcfe64768b40477ffc2053e27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.056690 746 tablet_service.cc:1467] Processing CreateTablet for tablet 125b12a2f9164653be79d3e6d8313d36 (DEFAULT_TABLE table=default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.056793 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff1b3ee229c445b19405cf4a19cf8003. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.056927 744 tablet_service.cc:1467] Processing CreateTablet for tablet c5079f87652a4ea58b366d1f9448fd4e (DEFAULT_TABLE table=default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.057497 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 125b12a2f9164653be79d3e6d8313d36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.057650 757 tablet_service.cc:1467] Processing CreateTablet for tablet f69b7a0f52304ad9951a07b3ab0372e7 (DEFAULT_TABLE table=default.Table_73e52096e12e47bab61cbfd2c72c70b1 [id=5424d9b98fae48949ed175c5498d24e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.058240 758 tablet_service.cc:1467] Processing CreateTablet for tablet 9bbef42e11a843b18ed582ab4d04a92a (DEFAULT_TABLE table=default.Table_c816af31a1e945518fd886cbf8b8b885 [id=93ab7e5945514c0aa85b411f15483f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.058338 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f69b7a0f52304ad9951a07b3ab0372e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.059100 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbef42e11a843b18ed582ab4d04a92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.059644 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5079f87652a4ea58b366d1f9448fd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.060828 759 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d411f906b4d4eb68343bdcc493e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:34.060984 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 7e1954e7859f422f9690fe4a9384a8bb (table default.Table_829049fe22b74e62a87dd1bed65fb3fe [id=2e892eba68234ffca0be4ef3e49cf309], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 7e1954e7859f422f9690fe4a9384a8bb consensus state has no leader
I20250112 02:52:34.061026 743 tablet_service.cc:1467] Processing CreateTablet for tablet 058a9ca089cd4967887b0a1127c7d03f (DEFAULT_TABLE table=default.Table_a56064e2c6b449b7b1e32f2a69f2f7fe [id=448877a19b0d4ed2abbc3a796e40eaeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.061966 747 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3a4e76b724452c8b238aae0198e297 (DEFAULT_TABLE table=default.Table_add1776e850b4a1bbd63f05707b51873 [id=ec37a56b10ba407b86e851f07bddf528]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.062288 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d3a4e76b724452c8b238aae0198e297. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.062717 745 tablet_service.cc:1467] Processing CreateTablet for tablet c29a9ca43e034b43a00207e5847bbee3 (DEFAULT_TABLE table=default.Table_9284616ca0cd499cb51df0322db9948f [id=7f36d8ba6d864d32ad84be06ca6682ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.063529 1602 raft_consensus.cc:491] T fccff64af8ca4400afc116db98918119 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.063685 1602 raft_consensus.cc:513] T fccff64af8ca4400afc116db98918119 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.064206 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 058a9ca089cd4967887b0a1127c7d03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.064427 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_feaebd2f4cbb47b6b24ff79614994d6f" } new_table_name: "default.Table_37bd8ec69e20493b8d2685e143a51327" modify_external_catalogs: true
I20250112 02:52:34.064954 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c29a9ca43e034b43a00207e5847bbee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.065156 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_c0aaadb5e69e46e8983f4d289a44afe2" } modify_external_catalogs: true
I20250112 02:52:34.065708 759 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3d291ca3494f05ac23c7fe293e0bd9 (DEFAULT_TABLE table=default.Table_c11c44af47604f9589b15d5c6a97c255 [id=e80fb97e841a4e68a6ab992d897816c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.066274 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d3d291ca3494f05ac23c7fe293e0bd9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:34.066375 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d63d411f906b4d4eb68343bdcc493e05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d63d411f906b4d4eb68343bdcc493e05 already in progress: creating tablet
I20250112 02:52:34.067327 753 tablet_service.cc:1467] Processing CreateTablet for tablet 13578542e9e741e8817e0ee63b10f1a1 (DEFAULT_TABLE table=default.Table_63abe3ac694a4ed6bf4aef720b542e73 [id=a7ce19c248fe4ac6a6b436773d6c1b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.067660 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13578542e9e741e8817e0ee63b10f1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.067904 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_9dd5993a01b44a0abe33a5738c8d5c2b" } modify_external_catalogs: true
W20250112 02:52:34.068120 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 2a6650515ea540a8bbf21e2f88363360 consensus state has no leader
I20250112 02:52:34.068702 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_9dd5993a01b44a0abe33a5738c8d5c2b" } modify_external_catalogs: true
I20250112 02:52:34.069829 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_c0aaadb5e69e46e8983f4d289a44afe2" } modify_external_catalogs: true
W20250112 02:52:34.070255 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet ead501d76bb1412e9839fb36b9911e6f consensus state has no leader
W20250112 02:52:34.070711 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 5cc414d6420c41839621ab858067548b (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 5cc414d6420c41839621ab858067548b consensus state has no leader
I20250112 02:52:34.071545 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_0dec5de812ab4ea48f1b7a8f4cb0cc4f" } new_table_name: "default.Table_9b25aff63ebb4620a3efe624832af86c" modify_external_catalogs: true
I20250112 02:52:34.072329 1101 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:34.071529 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f22a7b1b804661af3b7e785a833739 with delete_type TABLET_DATA_DELETED (replaced by tablet 3d662e0ce13f447595e5f92b94de3018) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:34.073881 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_9487626cbda14256af1f17d2675b5d33 [id=485dd281891f45e8b407b306572d6e29], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet b9537fcd96eb41cc8f680075ec71137b consensus state has no leader
I20250112 02:52:34.075052 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_a56064e2c6b449b7b1e32f2a69f2f7fe" } modify_external_catalogs: true
I20250112 02:52:34.075244 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_a56064e2c6b449b7b1e32f2a69f2f7fe" } modify_external_catalogs: true
I20250112 02:52:34.075989 755 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5b3c712c0f448ea9b14ab93dd731ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.076110 1602 leader_election.cc:290] T fccff64af8ca4400afc116db98918119 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:34.076049 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccff64af8ca4400afc116db98918119" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:34.076257 1055 raft_consensus.cc:2463] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
W20250112 02:52:34.076406 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet fccff64af8ca4400afc116db98918119 (table default.Table_15bcb2c0f2a74458a1bed231fd3437fc [id=ca9c8281bf254f248076cb600e7c4f87], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet fccff64af8ca4400afc116db98918119 consensus state has no leader
I20250112 02:52:34.076512 2003 tablet_replica.cc:331] T 65f22a7b1b804661af3b7e785a833739 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:34.076735 2003 raft_consensus.cc:2238] T 65f22a7b1b804661af3b7e785a833739 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250112 02:52:34.076630 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet f924459d1150472aafb325a11825db23 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet f924459d1150472aafb325a11825db23 consensus state has no leader
W20250112 02:52:34.077173 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_feaebd2f4cbb47b6b24ff79614994d6f [id=63c20c66463d4a30b7e1558fecce2704], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 896908b2726c40f9aabdfc9a6b3283f7 consensus state has no leader
W20250112 02:52:34.077409 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_0dec5de812ab4ea48f1b7a8f4cb0cc4f [id=2f43ebd749304c0287769224ac6b12c7], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 5d0c3191973a477e949fbf9dfcca963f consensus state has no leader
W20250112 02:52:34.079115 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 82b351c563d749df806ab5d0331a4ef2 (table default.Table_4603068c01544e2e965dca5177e36b40 [id=8502f21e9b61436093db1241cfbb5ec9], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 82b351c563d749df806ab5d0331a4ef2 consensus state has no leader
W20250112 02:52:34.079536 1778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 02:52:34.080528 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 5804fc4130f84c69832c1022b7bd7fa5 consensus state has no leader
I20250112 02:52:34.080955 1173 tablet_bootstrap.cc:492] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
W20250112 02:52:34.081108 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet b8b64e4e64cc462891f1ee7a594d5026 consensus state has no leader
I20250112 02:52:34.081133 1173 ts_tablet_manager.cc:1397] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
W20250112 02:52:34.081738 1778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:52:34.081867 1778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 02:52:34.081959 1778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 02:52:34.081921 1173 raft_consensus.cc:357] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.082137 1173 raft_consensus.cc:383] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:34.082221 1173 raft_consensus.cc:738] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:34.082476 1173 consensus_queue.cc:260] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:34.083139 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fc52271535df47e282bf1625c2e8e041 has no consensus state
I20250112 02:52:34.083496 1173 ts_tablet_manager.cc:1428] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:34.083541 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_9487626cbda14256af1f17d2675b5d33" } new_table_name: "default.Table_b8e142e93c8d481798ca4063a3d26e82" modify_external_catalogs: true
I20250112 02:52:34.083755 1173 tablet_bootstrap.cc:492] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:34.084582 452 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet cf54f5185a0f4c70b9057309fca14e56 consensus state has no leader
I20250112 02:52:34.086041 755 tablet_service.cc:1514] Processing DeleteTablet for tablet aadb2ab1d2024f889841ac91e065e064 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.086071 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 416fb6820cd54e118950c68d381f51b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:34.085661 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet b7340a7a916c4b2fab32e0ba2d0ce322 consensus state has no leader
W20250112 02:52:34.086476 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet c73f2fa452b84be398476b8aba6c9a08 consensus state has no leader
I20250112 02:52:34.086792 1173 tablet_bootstrap.cc:654] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:34.087034 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 70715db0394d46d9b6b289160b00be7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:34.090167 452 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 47a0d939fdb94590a86389f1f4693e9a consensus state has no leader
W20250112 02:52:34.090526 452 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet edfd1ebe729d458ea31eb7feff7ee072 consensus state has no leader
W20250112 02:52:34.090694 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 37862023095644bab2403576e8cb8c58 consensus state has no leader
I20250112 02:52:34.096076 1028 tablet_service.cc:1467] Processing CreateTablet for tablet c5079f87652a4ea58b366d1f9448fd4e (DEFAULT_TABLE table=default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.096482 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5079f87652a4ea58b366d1f9448fd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.096716 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c639a2bcd4740e4be7b2b162004669a with delete_type TABLET_DATA_DELETED (replaced by tablet 5d0c3191973a477e949fbf9dfcca963f) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.096911 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 87759e8b99b04e628badc8f60cb38bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.096940 741 tablet_service.cc:1514] Processing DeleteTablet for tablet fe614eaeb793473d9b2da206b7345562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.097071 748 tablet_service.cc:1514] Processing DeleteTablet for tablet b9199b0d2b924a79938a979df4196bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.097218 755 tablet_service.cc:1514] Processing DeleteTablet for tablet bb217afc08304c0ea35dee3120347ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.096689 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 161fe26b9c0e4543b7aeb0021506d72a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.102272 1021 tablet_service.cc:1467] Processing CreateTablet for tablet ff1b3ee229c445b19405cf4a19cf8003 (DEFAULT_TABLE table=default.Table_6b8cb70a5e5b4519af5d247f6b5bb8b0 [id=a28f5cfbcfe64768b40477ffc2053e27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.102632 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff1b3ee229c445b19405cf4a19cf8003. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.102283 1027 tablet_service.cc:1467] Processing CreateTablet for tablet f69b7a0f52304ad9951a07b3ab0372e7 (DEFAULT_TABLE table=default.Table_73e52096e12e47bab61cbfd2c72c70b1 [id=5424d9b98fae48949ed175c5498d24e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.103232 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f69b7a0f52304ad9951a07b3ab0372e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.103281 2003 raft_consensus.cc:2267] T 65f22a7b1b804661af3b7e785a833739 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:34.107744 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 41e166b3f8414ababbd4a3f07fb90c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.112813 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0c2fad4f3243238641aae56b305bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.113981 1727 raft_consensus.cc:491] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.114120 1727 raft_consensus.cc:513] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.114748 1727 leader_election.cc:290] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:34.114867 1727 raft_consensus.cc:491] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.114971 1727 raft_consensus.cc:513] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.115401 1727 leader_election.cc:290] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:34.117739 2003 ts_tablet_manager.cc:1905] T 65f22a7b1b804661af3b7e785a833739 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:34.119818 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f4e4f1e451b48dc9cdd780300f6888c with delete_type TABLET_DATA_DELETED (replaced by tablet 2052b4536f9745f0a785f330d706c1cf) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.121790 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 42cb7e7a41634cf1bdb978ac9d2befcd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.123003 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f59617b34046568dd83bbb7cdc42d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.124938 1727 raft_consensus.cc:491] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.125066 1727 raft_consensus.cc:513] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.125674 1727 leader_election.cc:290] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:34.126015 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a8a178bafcb457f9b0e9587df560493 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:34.132010 452 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 consensus state has no leader
I20250112 02:52:34.132855 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c686fdc8896463c81c9f7c5f53a3da2 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c03b5e612d64314b210c11686fdfa2a) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:34.141613 452 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 9af55dab202444358ea1a0674d37946d (table default.Table_26dfe346379948b39f90b31653c2a463 [id=6c6f30e60e79451397841d090d11d8b7], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 9af55dab202444358ea1a0674d37946d consensus state has no leader
I20250112 02:52:34.143007 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 9bbef42e11a843b18ed582ab4d04a92a (DEFAULT_TABLE table=default.Table_c816af31a1e945518fd886cbf8b8b885 [id=93ab7e5945514c0aa85b411f15483f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.143445 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbef42e11a843b18ed582ab4d04a92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.143537 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 125b12a2f9164653be79d3e6d8313d36 (DEFAULT_TABLE table=default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.143944 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 125b12a2f9164653be79d3e6d8313d36. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:34.144150 452 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 777272e7159b438ebff4db93698411d0 consensus state has no leader
W20250112 02:52:34.144999 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 3d662e0ce13f447595e5f92b94de3018 consensus state has no leader
W20250112 02:52:34.145875 452 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet f31b777e7e0f43a28839d8c48c6a6590 (table default.Table_5fb592e41cc4406aa27546e7b41fdb88 [id=6402abce8d1d45c182cc989490c6a5f0], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet f31b777e7e0f43a28839d8c48c6a6590 consensus state has no leader
I20250112 02:52:34.147900 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 058a9ca089cd4967887b0a1127c7d03f (DEFAULT_TABLE table=default.Table_a56064e2c6b449b7b1e32f2a69f2f7fe [id=448877a19b0d4ed2abbc3a796e40eaeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.148257 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 058a9ca089cd4967887b0a1127c7d03f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:34.148397 452 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 6117dc8b32a64535a2f4167a03bed0d2 (table default.Table_365739770a11434dbf260ea912e7ea81 [id=868adb627654450c936ef198e49d853b], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 6117dc8b32a64535a2f4167a03bed0d2 consensus state has no leader
I20250112 02:52:34.149567 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 49bb6d20cfa0453e90e048825114525e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.163071 1030 tablet_service.cc:1467] Processing CreateTablet for tablet c29a9ca43e034b43a00207e5847bbee3 (DEFAULT_TABLE table=default.Table_9284616ca0cd499cb51df0322db9948f [id=7f36d8ba6d864d32ad84be06ca6682ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.170043 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3a4e76b724452c8b238aae0198e297 (DEFAULT_TABLE table=default.Table_add1776e850b4a1bbd63f05707b51873 [id=ec37a56b10ba407b86e851f07bddf528]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.174376 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3d291ca3494f05ac23c7fe293e0bd9 (DEFAULT_TABLE table=default.Table_c11c44af47604f9589b15d5c6a97c255 [id=e80fb97e841a4e68a6ab992d897816c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.174887 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 13578542e9e741e8817e0ee63b10f1a1 (DEFAULT_TABLE table=default.Table_63abe3ac694a4ed6bf4aef720b542e73 [id=a7ce19c248fe4ac6a6b436773d6c1b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.176864 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c29a9ca43e034b43a00207e5847bbee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.177053 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d3d291ca3494f05ac23c7fe293e0bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.177340 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13578542e9e741e8817e0ee63b10f1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.188648 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d3a4e76b724452c8b238aae0198e297. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.193240 1602 raft_consensus.cc:491] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.193250 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 61593969d2234ce3a6089027b067d922 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:34.199869 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 003a3b4705e943c6a900762ec11ddd47 (table default.Table_743b3f7029534564b35c0c137b35fd75 [id=b2fa3922f6d447769a8179052f9a1861], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 003a3b4705e943c6a900762ec11ddd47 consensus state has no leader
W20250112 02:52:34.200538 452 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 8ee98f055024429eafaa29bb961cab87 (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 8ee98f055024429eafaa29bb961cab87 consensus state has no leader
I20250112 02:52:34.028625 2024 ts_tablet_manager.cc:1918] T c04b64bc2f8348a086ed3c4e59f09b16 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250112 02:52:34.030171 991 leader_election.cc:343] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:34.205298 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 175070us
I20250112 02:52:34.206434 1727 raft_consensus.cc:491] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.206585 1727 raft_consensus.cc:513] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.207197 1727 leader_election.cc:290] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:34.207352 1727 raft_consensus.cc:491] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.207458 1727 raft_consensus.cc:513] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.209091 2024 log.cc:1198] T c04b64bc2f8348a086ed3c4e59f09b16 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/c04b64bc2f8348a086ed3c4e59f09b16
W20250112 02:52:34.210353 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 896908b2726c40f9aabdfc9a6b3283f7 consensus state has no leader
W20250112 02:52:34.210440 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d0c3191973a477e949fbf9dfcca963f consensus state has no leader
I20250112 02:52:34.210848 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_f38c3f9c44b741219f03cf46cfbc9937"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:34.211253 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet b8daf4ef3593410eae87672402292be6
W20250112 02:52:34.211910 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b9537fcd96eb41cc8f680075ec71137b consensus state has no leader
I20250112 02:52:34.212174 1602 raft_consensus.cc:513] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.212931 1602 leader_election.cc:290] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:34.213433 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 64707ee432fb4e96a3bbf7252dfa1813 with delete_type TABLET_DATA_DELETED (replaced by tablet b3c5a861a90840d0a33b780ec75bffd2) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.214179 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_e781b72b17d54bf8ba293cd013e5e14d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:34.214946 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e781b72b17d54bf8ba293cd013e5e14d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:34.211431 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f38c3f9c44b741219f03cf46cfbc9937 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:34.215665 2052 raft_consensus.cc:491] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.216048 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c10fad790046a88fbc8ebe0303c481" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:34.216208 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet bab4978edea0445fb0456e88e7fb4c44
W20250112 02:52:34.216228 989 leader_election.cc:343] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: a1a0a13696fc41ccb28218f1275a4b5d
I20250112 02:52:34.216331 989 leader_election.cc:304] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:34.216563 1877 raft_consensus.cc:2743] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:34.217034 989 leader_election.cc:304] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:34.217394 1877 raft_consensus.cc:2798] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:34.217538 1877 raft_consensus.cc:695] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:34.217806 1877 consensus_queue.cc:237] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.218140 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet b8daf4ef3593410eae87672402292be6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.218510 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet bab4978edea0445fb0456e88e7fb4c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.219318 759 tablet_service.cc:1514] Processing DeleteTablet for tablet f7cd0927c6a4401cbe69f55d4db9963a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.210848 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_2989b0fb83514c45ac9836e95e444f31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:34.225994 991 leader_election.cc:343] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: b2d8ea2ef18a46508371b4789ab6f4fa
W20250112 02:52:34.226182 991 leader_election.cc:343] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: 97b31ebe035c4552baaffc3133ec73c1
I20250112 02:52:34.225720 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 058a9ca089cd4967887b0a1127c7d03f
W20250112 02:52:34.226351 991 leader_election.cc:343] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: a47cdc208fbc431b8326364e84432cf5
I20250112 02:52:34.227680 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73136be25c79491784d3ea71aa9b27f2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:34.228338 991 leader_election.cc:343] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:34.228363 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2989b0fb83514c45ac9836e95e444f31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:34.228550 991 leader_election.cc:343] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: f6803d0f553b4204b415b2734286bf4a
I20250112 02:52:34.230835 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_b2879d6a85294514989d7b68d3ac97d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:34.231635 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2879d6a85294514989d7b68d3ac97d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:34.232534 1727 leader_election.cc:290] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:34.232684 2052 raft_consensus.cc:513] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.233201 2052 leader_election.cc:290] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:34.239271 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1e3af2883341e9b957f6ed748e604d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:34.240077 991 leader_election.cc:343] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.241642 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a12b225131b464494a9f02133d37aea with delete_type TABLET_DATA_DELETED (replaced by tablet d45fce86e02f46d4a57af718fa2a62c8) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.242763 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd498ccce484a89855c982762086890 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.243151 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 015412a5fa304453bd8d1be1c3cd7a46 with delete_type TABLET_DATA_DELETED (replaced by tablet f924459d1150472aafb325a11825db23) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.244001 748 tablet_service.cc:1514] Processing DeleteTablet for tablet b7bd96fb381f4e73a0081be3fc799c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.244342 748 tablet_service.cc:1514] Processing DeleteTablet for tablet ac716c6da01941a989754aac1ef430d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.245057 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d0f67141dc46b6a87ca0ad080d5bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.245430 750 tablet_service.cc:1514] Processing DeleteTablet for tablet d659c5a4f7a446c6aaf2283f65b45106 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.245747 744 tablet_service.cc:1514] Processing DeleteTablet for tablet ea33e5c1a1fb46599c5836a36b130051 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.246146 744 tablet_service.cc:1514] Processing DeleteTablet for tablet fb0aba0139cf479b80b32f3dfc936535 with delete_type TABLET_DATA_DELETED (replaced by tablet e2f1f1a76e5f4308bf56dd647f02c6d8) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.246693 741 tablet_service.cc:1514] Processing DeleteTablet for tablet f05c8426ea3d4fe89fbe6c88c688019f with delete_type TABLET_DATA_DELETED (replaced by tablet c5079f87652a4ea58b366d1f9448fd4e) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.246804 752 tablet_service.cc:1514] Processing DeleteTablet for tablet daef3a58605e40978f7389698b22265d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.246973 741 tablet_service.cc:1514] Processing DeleteTablet for tablet b2ccb88108f24e36ba4b7ebf36398950 with delete_type TABLET_DATA_DELETED (replaced by tablet f3771166b77e4f90aaa116ab1a9dc299) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.247087 752 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d84027ad6d40a5be5afcdd854431a5 with delete_type TABLET_DATA_DELETED (replaced by tablet fbe9e9624ede40c29ea852b2b5494965) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.247315 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b599e983ec04f53a440d1ab828d12ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.247712 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 9163878cfeb64693a9c5493e812546af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.248126 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd48d0e68014e1aa0c44620527baec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.248663 754 tablet_service.cc:1514] Processing DeleteTablet for tablet e5c10fad790046a88fbc8ebe0303c481 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.249095 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d5b3df88a0c45dba879b3d506710614 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.249464 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a34d03a80ae402db6607261a5c40b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.249769 1173 tablet_bootstrap.cc:492] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:34.249907 1173 ts_tablet_manager.cc:1397] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.166s user 0.002s sys 0.006s
I20250112 02:52:34.250834 1173 raft_consensus.cc:357] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.251088 1173 raft_consensus.cc:383] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:34.251189 1173 raft_consensus.cc:738] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:34.028653 1181 tablet_bootstrap.cc:492] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:34.051901 898 tablet_service.cc:1467] Processing CreateTablet for tablet c5079f87652a4ea58b366d1f9448fd4e (DEFAULT_TABLE table=default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.252779 2036 log.cc:1198] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/55fb1a197d7e40689ccad82a44492371
I20250112 02:52:34.252921 1181 ts_tablet_manager.cc:1397] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.232s user 0.002s sys 0.002s
I20250112 02:52:34.253131 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5079f87652a4ea58b366d1f9448fd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.253227 2036 ts_tablet_manager.cc:1939] T 55fb1a197d7e40689ccad82a44492371 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:34.253808 1181 raft_consensus.cc:357] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.254024 1181 raft_consensus.cc:383] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:34.254078 1181 raft_consensus.cc:738] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:34.254236 1181 consensus_queue.cc:260] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.254832 1181 ts_tablet_manager.cc:1428] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:34.255112 1181 tablet_bootstrap.cc:492] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:34.252660 757 tablet_service.cc:1514] Processing DeleteTablet for tablet cc62dcbf67e84e36ae1fbe127ccd4879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.257694 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1e3af2883341e9b957f6ed748e604d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.258533 898 tablet_service.cc:1514] Processing DeleteTablet for tablet d45fce86e02f46d4a57af718fa2a62c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 5804fc4130f84c69832c1022b7bd7fa5) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.053114 891 tablet_service.cc:1467] Processing CreateTablet for tablet ff1b3ee229c445b19405cf4a19cf8003 (DEFAULT_TABLE table=default.Table_6b8cb70a5e5b4519af5d247f6b5bb8b0 [id=a28f5cfbcfe64768b40477ffc2053e27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.258862 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff1b3ee229c445b19405cf4a19cf8003. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.259677 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 04f135b226774285a077e5806131d91b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.259752 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 352c87f1dc724e7b94bc2e64abfce8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.260540 760 tablet_service.cc:1514] Processing DeleteTablet for tablet a85300a9aa1c4c50a3957a004bea7933 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.053702 899 tablet_service.cc:1467] Processing CreateTablet for tablet 9bbef42e11a843b18ed582ab4d04a92a (DEFAULT_TABLE table=default.Table_c816af31a1e945518fd886cbf8b8b885 [id=93ab7e5945514c0aa85b411f15483f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.060199 893 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3a4e76b724452c8b238aae0198e297 (DEFAULT_TABLE table=default.Table_add1776e850b4a1bbd63f05707b51873 [id=ec37a56b10ba407b86e851f07bddf528]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.263549 2024 ts_tablet_manager.cc:1939] T c04b64bc2f8348a086ed3c4e59f09b16 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:34.263736 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbef42e11a843b18ed582ab4d04a92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.263916 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d3a4e76b724452c8b238aae0198e297. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.060179 889 tablet_service.cc:1467] Processing CreateTablet for tablet 058a9ca089cd4967887b0a1127c7d03f (DEFAULT_TABLE table=default.Table_a56064e2c6b449b7b1e32f2a69f2f7fe [id=448877a19b0d4ed2abbc3a796e40eaeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.264084 1173 consensus_queue.cc:260] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.264221 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 058a9ca089cd4967887b0a1127c7d03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.265043 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 058a9ca089cd4967887b0a1127c7d03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.267293 1173 ts_tablet_manager.cc:1428] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.016s user 0.000s sys 0.003s
I20250112 02:52:34.060642 890 tablet_service.cc:1467] Processing CreateTablet for tablet 125b12a2f9164653be79d3e6d8313d36 (DEFAULT_TABLE table=default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.066154 887 tablet_service.cc:1467] Processing CreateTablet for tablet c29a9ca43e034b43a00207e5847bbee3 (DEFAULT_TABLE table=default.Table_9284616ca0cd499cb51df0322db9948f [id=7f36d8ba6d864d32ad84be06ca6682ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.267693 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 125b12a2f9164653be79d3e6d8313d36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.268190 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c29a9ca43e034b43a00207e5847bbee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.066567 895 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3d291ca3494f05ac23c7fe293e0bd9 (DEFAULT_TABLE table=default.Table_c11c44af47604f9589b15d5c6a97c255 [id=e80fb97e841a4e68a6ab992d897816c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.271678 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d3d291ca3494f05ac23c7fe293e0bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.272228 759 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a3486ef011492c8a1df8d6278adedf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.267583 1173 tablet_bootstrap.cc:492] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:34.273164 893 tablet_service.cc:1514] Processing DeleteTablet for tablet f234256b343f447dadfc3d1936a824ee with delete_type TABLET_DATA_DELETED (replaced by tablet 82b351c563d749df806ab5d0331a4ef2) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.275286 887 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a7281c0c09442ebd0cc13b8496c86e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.066799 897 tablet_service.cc:1467] Processing CreateTablet for tablet 13578542e9e741e8817e0ee63b10f1a1 (DEFAULT_TABLE table=default.Table_63abe3ac694a4ed6bf4aef720b542e73 [id=a7ce19c248fe4ac6a6b436773d6c1b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.275424 887 tablet_service.cc:1514] Processing DeleteTablet for tablet a12c5d58e6c246c983e8a533906ce77d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.275564 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb76746b3a846c3aba531c04bf63966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.070035 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccff64af8ca4400afc116db98918119" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:34.275684 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 73136be25c79491784d3ea71aa9b27f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.275787 917 raft_consensus.cc:2463] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:34.275825 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 352c87f1dc724e7b94bc2e64abfce8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.081247 2039 raft_consensus.cc:491] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.275945 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b184806b4884cd3b0a002905a1e57af with delete_type TABLET_DATA_DELETED (replaced by tablet 47a0d939fdb94590a86389f1f4693e9a) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.276069 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a34d03a80ae402db6607261a5c40b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.276006 2039 raft_consensus.cc:513] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.276202 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d5b3df88a0c45dba879b3d506710614 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.276350 887 tablet_service.cc:1514] Processing DeleteTablet for tablet a85300a9aa1c4c50a3957a004bea7933 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.276465 887 tablet_service.cc:1514] Processing DeleteTablet for tablet e5c10fad790046a88fbc8ebe0303c481 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.276582 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd48d0e68014e1aa0c44620527baec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.276621 2039 leader_election.cc:290] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:34.082876 885 tablet_service.cc:1514] Processing DeleteTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.276705 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b599e983ec04f53a440d1ab828d12ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.276790 885 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d84027ad6d40a5be5afcdd854431a5 with delete_type TABLET_DATA_DELETED (replaced by tablet fbe9e9624ede40c29ea852b2b5494965) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.275005 2003 ts_tablet_manager.cc:1918] T 65f22a7b1b804661af3b7e785a833739 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:34.276824 887 tablet_service.cc:1514] Processing DeleteTablet for tablet b2ccb88108f24e36ba4b7ebf36398950 with delete_type TABLET_DATA_DELETED (replaced by tablet f3771166b77e4f90aaa116ab1a9dc299) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.276926 885 tablet_service.cc:1514] Processing DeleteTablet for tablet bb217afc08304c0ea35dee3120347ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.276955 887 tablet_service.cc:1514] Processing DeleteTablet for tablet ea33e5c1a1fb46599c5836a36b130051 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.277040 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 416fb6820cd54e118950c68d381f51b2) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.277052 887 tablet_service.cc:1514] Processing DeleteTablet for tablet ac716c6da01941a989754aac1ef430d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.277173 885 tablet_service.cc:1514] Processing DeleteTablet for tablet bab4978edea0445fb0456e88e7fb4c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.277184 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 058a9ca089cd4967887b0a1127c7d03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.277289 885 tablet_service.cc:1514] Processing DeleteTablet for tablet b8daf4ef3593410eae87672402292be6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.277289 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 64707ee432fb4e96a3bbf7252dfa1813 with delete_type TABLET_DATA_DELETED (replaced by tablet b3c5a861a90840d0a33b780ec75bffd2) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.277433 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 015412a5fa304453bd8d1be1c3cd7a46 with delete_type TABLET_DATA_DELETED (replaced by tablet f924459d1150472aafb325a11825db23) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.277537 885 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d411f906b4d4eb68343bdcc493e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.277658 885 tablet_service.cc:1514] Processing DeleteTablet for tablet f05c8426ea3d4fe89fbe6c88c688019f with delete_type TABLET_DATA_DELETED (replaced by tablet c5079f87652a4ea58b366d1f9448fd4e) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.277778 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a12b225131b464494a9f02133d37aea with delete_type TABLET_DATA_DELETED (replaced by tablet d45fce86e02f46d4a57af718fa2a62c8) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.277917 885 tablet_service.cc:1514] Processing DeleteTablet for tablet d4abf59d96414e5faa7cf75ba133832c with delete_type TABLET_DATA_DELETED (replaced by tablet 5a3af0ea4e954053ae89814f3e74e6b7) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.278041 885 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a3486ef011492c8a1df8d6278adedf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.278164 885 tablet_service.cc:1514] Processing DeleteTablet for tablet cc62dcbf67e84e36ae1fbe127ccd4879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.278271 885 tablet_service.cc:1514] Processing DeleteTablet for tablet d659c5a4f7a446c6aaf2283f65b45106 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.278419 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d0f67141dc46b6a87ca0ad080d5bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.278529 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a8a178bafcb457f9b0e9587df560493 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.278623 885 tablet_service.cc:1514] Processing DeleteTablet for tablet fe614eaeb793473d9b2da206b7345562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.278714 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 41e166b3f8414ababbd4a3f07fb90c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.278849 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 04f135b226774285a077e5806131d91b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.278978 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1e3af2883341e9b957f6ed748e604d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.279090 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 49bb6d20cfa0453e90e048825114525e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.281555 759 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b184806b4884cd3b0a002905a1e57af with delete_type TABLET_DATA_DELETED (replaced by tablet 47a0d939fdb94590a86389f1f4693e9a) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.083535 888 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5b3c712c0f448ea9b14ab93dd731ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.084306 884 tablet_service.cc:1514] Processing DeleteTablet for tablet b1d9fae290ec4281839991f9d40b105c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.085166 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c821ea3ca4946c290c17135366e0cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.091365 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 06edfca60eed4dd3a2dee7b662b0970c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.275684 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13578542e9e741e8817e0ee63b10f1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.092296 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.093421 881 tablet_service.cc:1514] Processing DeleteTablet for tablet b9c006281dc949119951c74e23167d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.105233 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 2730634a5cfd4c2fb5c4e30100c556d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.105381 880 tablet_service.cc:1514] Processing DeleteTablet for tablet b61f16f3d601436bb4b32344de28c1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.107312 2032 raft_consensus.cc:491] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.283620 2032 raft_consensus.cc:513] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:34.284099 2032 leader_election.cc:290] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:34.124681 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a18132549374218893f35a4e4226b72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.233562 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd48d0e68014e1aa0c44620527baec9" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:34.284379 919 raft_consensus.cc:2388] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 289afc116aa74408aa5bff74e11c50d8 in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
W20250112 02:52:34.244788 854 leader_election.cc:343] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.284708 854 leader_election.cc:304] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:34.285583 854 leader_election.cc:343] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.285693 854 leader_election.cc:304] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:34.285913 2039 raft_consensus.cc:2743] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:34.286053 2039 raft_consensus.cc:2743] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:34.286460 854 leader_election.cc:343] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.286563 854 leader_election.cc:304] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:34.286780 2039 raft_consensus.cc:2743] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:34.287118 2036 tablet_replica.cc:331] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:34.287283 2036 raft_consensus.cc:2238] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:34.052546 892 tablet_service.cc:1467] Processing CreateTablet for tablet f69b7a0f52304ad9951a07b3ab0372e7 (DEFAULT_TABLE table=default.Table_73e52096e12e47bab61cbfd2c72c70b1 [id=5424d9b98fae48949ed175c5498d24e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.287724 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f69b7a0f52304ad9951a07b3ab0372e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.288762 2003 log.cc:1198] T 65f22a7b1b804661af3b7e785a833739 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/65f22a7b1b804661af3b7e785a833739
I20250112 02:52:34.289058 755 tablet_service.cc:1514] Processing DeleteTablet for tablet c7c530aa0aa64e47ae27d61f341d2bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.289237 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 416fb6820cd54e118950c68d381f51b2) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:34.287293 854 leader_election.cc:343] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.291615 854 leader_election.cc:304] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:34.291849 2039 raft_consensus.cc:2743] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:34.292909 2039 raft_consensus.cc:491] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.293046 2039 raft_consensus.cc:513] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.293586 2039 leader_election.cc:290] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:34.293766 2039 raft_consensus.cc:491] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.293877 2039 raft_consensus.cc:513] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.294198 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 73136be25c79491784d3ea71aa9b27f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.294354 2039 leader_election.cc:290] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:34.299249 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f234256b343f447dadfc3d1936a824ee" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:34.300076 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb76746b3a846c3aba531c04bf63966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.301749 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0dc78d722ac4687bfb3dce94d6db7f0" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:34.302333 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c5a861a90840d0a33b780ec75bffd2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:34.304726 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb217afc08304c0ea35dee3120347ba1" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:34.304909 917 raft_consensus.cc:3054] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:34.307626 917 raft_consensus.cc:2463] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:34.308179 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccff64af8ca4400afc116db98918119" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:34.308416 917 raft_consensus.cc:2463] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:34.309141 744 tablet_service.cc:1514] Processing DeleteTablet for tablet a12c5d58e6c246c983e8a533906ce77d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.309321 1177 tablet_bootstrap.cc:492] T 1a1e3af2883341e9b957f6ed748e604d P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:34.309427 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b70dce4d1d4e29997a18297ac0f82a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:34.309561 917 raft_consensus.cc:2463] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:34.310547 744 tablet_service.cc:1514] Processing DeleteTablet for tablet d4abf59d96414e5faa7cf75ba133832c with delete_type TABLET_DATA_DELETED (replaced by tablet 5a3af0ea4e954053ae89814f3e74e6b7) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.310575 1177 ts_tablet_manager.cc:1397] T 1a1e3af2883341e9b957f6ed748e604d P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.277s user 0.004s sys 0.035s
W20250112 02:52:34.312805 439 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:38771, user_credentials={real_user=slave}} blocked reactor thread for 80309.9us
I20250112 02:52:34.314164 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 55fb1a197d7e40689ccad82a44492371 (table default.Table_f3695b00a89b4353903587a6c1418d8c [id=d1781738b8a74f139b09863918cbb21d]) successfully deleted
W20250112 02:52:34.314422 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 9af55dab202444358ea1a0674d37946d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9af55dab202444358ea1a0674d37946d already in progress: creating tablet
I20250112 02:52:34.315188 2036 pending_rounds.cc:62] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d: Trying to abort 1 pending ops.
I20250112 02:52:34.315274 2036 pending_rounds.cc:69] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7113319850352709632 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "c04b64bc2f8348a086ed3c4e59f09b16" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 2 new_table_name: "default.Table_c3651f53d9004655b02efca2428b3ead" dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" new_extra_config { } }
W20250112 02:52:34.315659 2036 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38786: Aborted: Op aborted
I20250112 02:52:34.315963 2036 raft_consensus.cc:2267] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:34.316534 440 catalog_manager.cc:4951] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): alter failed for tablet c04b64bc2f8348a086ed3c4e59f09b16 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]): Aborted: Op aborted
I20250112 02:52:34.317454 1181 tablet_bootstrap.cc:654] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:34.317469 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73136be25c79491784d3ea71aa9b27f2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:34.317694 917 raft_consensus.cc:2463] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:34.318926 1173 tablet_bootstrap.cc:654] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:34.321516 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1e3af2883341e9b957f6ed748e604d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:34.321681 917 raft_consensus.cc:2463] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:34.322953 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_f3695b00a89b4353903587a6c1418d8c" } modify_external_catalogs: true
I20250112 02:52:34.323134 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_f3695b00a89b4353903587a6c1418d8c" } modify_external_catalogs: true
I20250112 02:52:34.324029 1877 raft_consensus.cc:491] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.324200 1877 raft_consensus.cc:513] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.325044 1877 leader_election.cc:290] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:34.333374 1173 tablet_bootstrap.cc:492] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:34.333572 1173 ts_tablet_manager.cc:1397] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.066s user 0.001s sys 0.003s
I20250112 02:52:34.334381 1173 raft_consensus.cc:357] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.335726 1173 raft_consensus.cc:383] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:34.335841 1173 raft_consensus.cc:738] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:34.336128 1173 consensus_queue.cc:260] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.337242 1877 raft_consensus.cc:491] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.337399 1877 raft_consensus.cc:513] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.338023 1877 leader_election.cc:290] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:34.338663 1173 ts_tablet_manager.cc:1428] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 02:52:34.339598 1877 consensus_queue.cc:1035] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:52:34.340085 854 leader_election.cc:343] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.340093 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223eb8b4d1024985844867d5641600d1" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:34.340198 854 leader_election.cc:304] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:34.340574 2114 raft_consensus.cc:2743] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:34.340787 854 leader_election.cc:343] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:34.340981 854 leader_election.cc:343] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.341056 2003 ts_tablet_manager.cc:1939] T 65f22a7b1b804661af3b7e785a833739 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:34.341513 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a7281c0c09442ebd0cc13b8496c86e" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:34.342152 989 leader_election.cc:343] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: aba1744138fe42ae9d097db01cac083d
I20250112 02:52:34.342883 989 leader_election.cc:304] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:34.343643 989 leader_election.cc:304] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:34.345315 1727 raft_consensus.cc:2798] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:34.345058 1877 raft_consensus.cc:2743] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:34.345436 1727 raft_consensus.cc:491] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:34.345552 1727 raft_consensus.cc:3054] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:34.347553 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet c04b64bc2f8348a086ed3c4e59f09b16 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]) successfully deleted
I20250112 02:52:34.344993 2036 ts_tablet_manager.cc:1905] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:34.351349 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b8daf4ef3593410eae87672402292be6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8daf4ef3593410eae87672402292be6 already in progress: creating tablet
W20250112 02:52:34.351652 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet bab4978edea0445fb0456e88e7fb4c44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bab4978edea0445fb0456e88e7fb4c44 already in progress: creating tablet
I20250112 02:52:34.345530 1181 tablet_bootstrap.cc:492] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:34.352321 1181 ts_tablet_manager.cc:1397] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.008s
I20250112 02:52:34.352619 1727 raft_consensus.cc:513] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.352927 1181 raft_consensus.cc:357] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.353102 1181 raft_consensus.cc:383] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:34.353152 1181 raft_consensus.cc:738] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:34.353310 1181 consensus_queue.cc:260] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.353569 1727 leader_election.cc:290] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:34.354220 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0dc78d722ac4687bfb3dce94d6db7f0" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:34.354456 989 leader_election.cc:304] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
W20250112 02:52:34.354935 989 leader_election.cc:343] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: a47cdc208fbc431b8326364e84432cf5
I20250112 02:52:34.355252 989 leader_election.cc:304] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:34.355139 1877 raft_consensus.cc:2798] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:34.355548 1877 raft_consensus.cc:491] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:34.355671 1877 raft_consensus.cc:3054] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:34.356001 2056 raft_consensus.cc:2743] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:34.356932 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccff64af8ca4400afc116db98918119" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:34.357120 774 raft_consensus.cc:3054] T fccff64af8ca4400afc116db98918119 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:34.358600 1877 raft_consensus.cc:513] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.359409 1877 leader_election.cc:290] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:34.359987 991 leader_election.cc:343] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: 2fdeaf5fa8cc4a2b97b2178d8ecbad69
W20250112 02:52:34.361900 854 leader_election.cc:343] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:34.362072 854 leader_election.cc:343] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.364282 2036 ts_tablet_manager.cc:1918] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20250112 02:52:34.364622 989 leader_election.cc:343] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 97b31ebe035c4552baaffc3133ec73c1
I20250112 02:52:34.364835 989 leader_election.cc:304] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
W20250112 02:52:34.365347 989 leader_election.cc:343] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: b2d8ea2ef18a46508371b4789ab6f4fa
I20250112 02:52:34.365689 989 leader_election.cc:304] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:34.365561 1727 raft_consensus.cc:2743] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:34.366137 1727 raft_consensus.cc:2743] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:34.366662 989 leader_election.cc:304] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:34.367012 1727 raft_consensus.cc:2798] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:34.367141 1727 raft_consensus.cc:491] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:34.367297 1727 raft_consensus.cc:3054] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:34.368402 713 leader_election.cc:304] T fccff64af8ca4400afc116db98918119 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:34.347350 1778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42341,127.0.104.61:38771,127.0.104.60:39987
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:38771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=33823
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4135
ASAN enabled
W20250112 02:52:34.369266 440 catalog_manager.cc:4945] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): alter failed for tablet c04b64bc2f8348a086ed3c4e59f09b16 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]),no further retry: Illegal state: Tablet not RUNNING: STOPPED
I20250112 02:52:34.369690 751 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a7281c0c09442ebd0cc13b8496c86e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.370431 1778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:52:34.372227 1727 raft_consensus.cc:513] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.373196 1727 leader_election.cc:290] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:34.373272 454 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:37987 (request call id 250) took 1336 ms (client timeout 1000 ms). Trace:
W20250112 02:52:34.373420 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 058a9ca089cd4967887b0a1127c7d03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 058a9ca089cd4967887b0a1127c7d03f already in progress: creating tablet
I20250112 02:52:34.373567 1778 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 02:52:34.373647 989 leader_election.cc:343] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: f6803d0f553b4204b415b2734286bf4a
I20250112 02:52:34.373760 989 leader_election.cc:304] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:34.374173 989 leader_election.cc:304] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:34.374867 2056 raft_consensus.cc:2743] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:34.375074 2056 raft_consensus.cc:2798] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:34.375211 2056 raft_consensus.cc:491] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:34.375296 2056 raft_consensus.cc:3054] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:34.375344 774 raft_consensus.cc:2463] T fccff64af8ca4400afc116db98918119 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:34.375380 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a18132549374218893f35a4e4226b72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:34.376113 991 leader_election.cc:343] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: afa2f8aabe5e42ccbc15b193fdc03563
I20250112 02:52:34.376308 991 leader_election.cc:304] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
W20250112 02:52:34.374622 454 rpcz_store.cc:269] 0112 02:52:33.036866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:33.036913 (+ 47us) service_pool.cc:225] Handling call
0112 02:52:34.373236 (+1336323us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:33.778244 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:33.778335 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:33.778341 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:33.778344 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:33.778439 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:33.778448 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:33.778453 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:33.778533 (+ 80us) tablet.cc:698] Row locks acquired
0112 02:52:33.778535 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:33.778562 (+ 27us) write_op.cc:270] Start()
0112 02:52:33.778587 (+ 25us) write_op.cc:276] Timestamp: P: 1736650353778481 usec, L: 0
0112 02:52:33.778590 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:33.778719 (+ 129us) log.cc:844] Serialized 901 byte log entry
0112 02:52:33.782095 (+ 3376us) op_driver.cc:464] REPLICATION: finished
0112 02:52:33.782696 (+ 601us) write_op.cc:301] APPLY: starting
0112 02:52:33.782708 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:33.782739 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:33.782772 (+ 33us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:33.782850 (+ 78us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:33.782884 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:33.782891 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:33.782950 (+ 59us) log.cc:844] Serialized 29 byte log entry
0112 02:52:33.782995 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:33.783013 (+ 18us) write_op.cc:454] Released schema lock
0112 02:52:33.783020 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:33.783081 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:33.783085 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":19171,"reported_tablets":176,"child_traces":[["op",{"apply.queue_time_us":564,"apply.run_cpu_time_us":381,"apply.run_wall_time_us":382,"num_ops":2,"prepare.queue_time_us":497,"prepare.run_cpu_time_us":690,"prepare.run_wall_time_us":1910,"raft.queue_time_us":1614,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":3490,"thread_start_us":1335,"threads_started":1}]]}
I20250112 02:52:34.378337 2056 raft_consensus.cc:513] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.378913 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_9afc6941825a45bc8f98bf373ff5f27c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:34.379063 2056 leader_election.cc:290] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:34.379177 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_743b3f7029534564b35c0c137b35fd75" } modify_external_catalogs: true
I20250112 02:52:34.379283 2056 raft_consensus.cc:2798] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:34.380017 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_743b3f7029534564b35c0c137b35fd75" } modify_external_catalogs: true
I20250112 02:52:34.380694 2056 raft_consensus.cc:695] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:34.381168 2056 consensus_queue.cc:237] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:34.388799 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9afc6941825a45bc8f98bf373ff5f27c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:34.389103 1173 tablet_bootstrap.cc:492] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:34.391021 1173 tablet_bootstrap.cc:654] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:34.368428 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 161fe26b9c0e4543b7aeb0021506d72a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.368543 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 416fb6820cd54e118950c68d381f51b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.394476 2036 log.cc:1198] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/c04b64bc2f8348a086ed3c4e59f09b16
I20250112 02:52:34.394917 965 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:34.395072 2036 ts_tablet_manager.cc:1939] T c04b64bc2f8348a086ed3c4e59f09b16 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:34.395602 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet c04b64bc2f8348a086ed3c4e59f09b16 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]) successfully deleted
W20250112 02:52:34.391875 989 leader_election.cc:343] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: afa2f8aabe5e42ccbc15b193fdc03563
I20250112 02:52:34.397470 989 leader_election.cc:304] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:34.397703 2052 raft_consensus.cc:2743] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:34.398088 989 leader_election.cc:343] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 2fdeaf5fa8cc4a2b97b2178d8ecbad69
I20250112 02:52:34.398249 989 leader_election.cc:304] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:34.398483 2052 raft_consensus.cc:2743] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:34.370733 1181 ts_tablet_manager.cc:1428] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.018s user 0.000s sys 0.001s
I20250112 02:52:34.399657 2028 tablet.cc:1721] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:34.389811 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f4e4f1e451b48dc9cdd780300f6888c with delete_type TABLET_DATA_DELETED (replaced by tablet 2052b4536f9745f0a785f330d706c1cf) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.401311 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 70715db0394d46d9b6b289160b00be7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.401315 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccff64af8ca4400afc116db98918119" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:34.402086 2036 tablet_replica.cc:331] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:34.402544 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5b3c712c0f448ea9b14ab93dd731ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.402161 918 raft_consensus.cc:3054] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:34.400427 1181 tablet_bootstrap.cc:492] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:34.402683 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b70dce4d1d4e29997a18297ac0f82a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:34.403506 919 raft_consensus.cc:3054] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:34.403647 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet b1d9fae290ec4281839991f9d40b105c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.403821 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 87759e8b99b04e628badc8f60cb38bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.404119 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c639a2bcd4740e4be7b2b162004669a with delete_type TABLET_DATA_DELETED (replaced by tablet 5d0c3191973a477e949fbf9dfcca963f) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.404369 899 tablet_service.cc:1514] Processing DeleteTablet for tablet f7cd0927c6a4401cbe69f55d4db9963a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.404634 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 61593969d2234ce3a6089027b067d922 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.404875 899 tablet_service.cc:1514] Processing DeleteTablet for tablet daef3a58605e40978f7389698b22265d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:38786
W20250112 02:52:34.405894 452 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:58959 (request call id 236) took 1383 ms (client timeout 1000 ms). Trace:
I20250112 02:52:34.406442 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_6730101812ca4bca8e85617d952cc549"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:34.406999 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6730101812ca4bca8e85617d952cc549 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:34.407773 919 raft_consensus.cc:2463] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:34.404302 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 70715db0394d46d9b6b289160b00be7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.410166 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet d659c5a4f7a446c6aaf2283f65b45106 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.410895 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d0f67141dc46b6a87ca0ad080d5bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.412253 1173 tablet_bootstrap.cc:492] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:34.414120 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_b645d30cd43d4f09a3ac34db205a80f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:34.414642 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b645d30cd43d4f09a3ac34db205a80f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:34.412708 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 416fb6820cd54e118950c68d381f51b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.413471 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 161fe26b9c0e4543b7aeb0021506d72a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.414155 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet b9199b0d2b924a79938a979df4196bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.415230 1173 ts_tablet_manager.cc:1397] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.076s user 0.001s sys 0.002s
I20250112 02:52:34.415239 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_75f163a2f4aa48df95c9bafd3b311f4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:34.415656 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75f163a2f4aa48df95c9bafd3b311f4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:34.415450 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet cb724feccc7f4fbc846fb6fa61be3385 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.416271 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1e3af2883341e9b957f6ed748e604d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.417559 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet c7c530aa0aa64e47ae27d61f341d2bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.417614 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_39f67a35c5c640dd8cbbcea89df5e4fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:34.418104 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39f67a35c5c640dd8cbbcea89df5e4fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:34.418776 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0c2fad4f3243238641aae56b305bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.419684 1173 raft_consensus.cc:357] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:34.419872 452 rpcz_store.cc:269] 0112 02:52:33.022207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:33.022422 (+ 215us) service_pool.cc:225] Handling call
0112 02:52:34.405830 (+1383408us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:33.708869 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:33.708989 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:33.708999 (+ 10us) write_op.cc:435] Acquired schema lock
0112 02:52:33.709002 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:33.709457 (+ 455us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:33.709473 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:33.709478 (+ 5us) tablet.cc:682] Acquiring locks for 16 operations
0112 02:52:33.709746 (+ 268us) tablet.cc:698] Row locks acquired
0112 02:52:33.709750 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:33.709790 (+ 40us) write_op.cc:270] Start()
0112 02:52:33.709825 (+ 35us) write_op.cc:276] Timestamp: P: 1736650353709719 usec, L: 0
0112 02:52:33.709831 (+ 6us) op_driver.cc:348] REPLICATION: starting
0112 02:52:33.710075 (+ 244us) log.cc:844] Serialized 6138 byte log entry
0112 02:52:33.716783 (+ 6708us) op_driver.cc:464] REPLICATION: finished
0112 02:52:33.716970 (+ 187us) write_op.cc:301] APPLY: starting
0112 02:52:33.716989 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:33.717157 (+ 168us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:33.717192 (+ 35us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:33.717583 (+ 391us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:33.717643 (+ 60us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
0112 02:52:33.717655 (+ 12us) write_op.cc:312] APPLY: finished
0112 02:52:33.717796 (+ 141us) log.cc:844] Serialized 113 byte log entry
0112 02:52:33.717872 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:33.717921 (+ 49us) write_op.cc:454] Released schema lock
0112 02:52:33.717932 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:33.718036 (+ 104us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:33.718042 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":164,"child_traces":[["op",{"apply.queue_time_us":126,"apply.run_cpu_time_us":1044,"apply.run_wall_time_us":1043,"num_ops":16,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":1324,"prepare.run_wall_time_us":1325,"raft.queue_time_us":1305,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":284,"replication_time_us":6917}]]}
I20250112 02:52:34.420084 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet aadb2ab1d2024f889841ac91e065e064 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.420395 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a0a13696fc41ccb28218f1275a4b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.421670 2032 raft_consensus.cc:491] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.420702 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 41e166b3f8414ababbd4a3f07fb90c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.421844 2032 raft_consensus.cc:513] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.422041 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73136be25c79491784d3ea71aa9b27f2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:34.422253 916 raft_consensus.cc:3054] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:34.422411 2032 leader_election.cc:290] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:34.425223 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_23c987a24fed4ceeb90fd25f0367b08f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:34.425845 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23c987a24fed4ceeb90fd25f0367b08f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:34.369531 1177 raft_consensus.cc:357] T 1a1e3af2883341e9b957f6ed748e604d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.426251 2071 tablet.cc:1721] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:34.421005 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet a47cdc208fbc431b8326364e84432cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.427268 916 raft_consensus.cc:2463] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:34.421337 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet fe614eaeb793473d9b2da206b7345562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.422062 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1e3af2883341e9b957f6ed748e604d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:34.427973 919 raft_consensus.cc:3054] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:34.429699 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_63649e9607394f4bb72a3dd6e6d0dd69" } new_table_name: "default.Table_57b3861e9f704dc09d466241c1926fc1" modify_external_catalogs: true
I20250112 02:52:34.431264 919 raft_consensus.cc:2463] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:34.431699 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet b2d8ea2ef18a46508371b4789ab6f4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.431936 1173 raft_consensus.cc:383] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:34.432303 1173 raft_consensus.cc:738] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:34.432554 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f59617b34046568dd83bbb7cdc42d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.433122 1173 consensus_queue.cc:260] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.436820 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_8f55d3bee66c40239788d75612403ca6" } new_table_name: "default.Table_f20419242ba541528d0cf9ec37ac4e92" modify_external_catalogs: true
W20250112 02:52:34.438361 439 consensus_peers.cc:487] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e -> Peer f4a49a5756b143fcb767c3f41463f8d9 (127.0.104.61:38771): Couldn't send request to peer f4a49a5756b143fcb767c3f41463f8d9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250112 02:52:34.437871 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 42cb7e7a41634cf1bdb978ac9d2befcd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.438750 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f4e4f1e451b48dc9cdd780300f6888c with delete_type TABLET_DATA_DELETED (replaced by tablet 2052b4536f9745f0a785f330d706c1cf) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.439927 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet ac716c6da01941a989754aac1ef430d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.441049 2056 raft_consensus.cc:491] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.441291 2056 raft_consensus.cc:513] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.442361 2056 leader_election.cc:290] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:34.442584 454 master_service.cc:946] ReplaceTablet: received request to replace tablet aadb2ab1d2024f889841ac91e065e064 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:34.443239 454 master_service.cc:946] ReplaceTablet: received request to replace tablet fe614eaeb793473d9b2da206b7345562 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:34.443933 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 62f4bd2df0ce45dabc55373f1154d3ac from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:34.444586 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 8ee98f055024429eafaa29bb961cab87 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:34.445045 749 tablet_service.cc:1514] Processing DeleteTablet for tablet d45fce86e02f46d4a57af718fa2a62c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 5804fc4130f84c69832c1022b7bd7fa5) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.445554 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a3486ef011492c8a1df8d6278adedf" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:34.445756 2024 tablet_replica.cc:331] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:34.446343 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223eb8b4d1024985844867d5641600d1" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:34.446555 2024 raft_consensus.cc:2238] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:34.446668 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c5a861a90840d0a33b780ec75bffd2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:34.446820 2024 pending_rounds.cc:62] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6: Trying to abort 1 pending ops.
I20250112 02:52:34.446935 2024 pending_rounds.cc:69] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319848848048128 op_type: NO_OP noop_request { }
I20250112 02:52:34.447060 2024 raft_consensus.cc:2883] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:34.447187 2024 raft_consensus.cc:2267] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:34.447966 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 2730634a5cfd4c2fb5c4e30100c556d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.448166 2024 ts_tablet_manager.cc:1905] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:34.448469 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73136be25c79491784d3ea71aa9b27f2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:34.448751 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b70dce4d1d4e29997a18297ac0f82a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:34.449113 1177 raft_consensus.cc:383] T 1a1e3af2883341e9b957f6ed748e604d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:34.449203 1177 raft_consensus.cc:738] T 1a1e3af2883341e9b957f6ed748e604d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:34.449427 1177 consensus_queue.cc:260] T 1a1e3af2883341e9b957f6ed748e604d P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.449961 1177 ts_tablet_manager.cc:1428] T 1a1e3af2883341e9b957f6ed748e604d P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.139s user 0.001s sys 0.001s
I20250112 02:52:34.450263 1177 tablet_bootstrap.cc:492] T 89b70dce4d1d4e29997a18297ac0f82a P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:34.451619 751 tablet_service.cc:1514] Processing DeleteTablet for tablet b8daf4ef3593410eae87672402292be6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.452261 1177 tablet_bootstrap.cc:654] T 89b70dce4d1d4e29997a18297ac0f82a P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:34.453184 1718 raft_consensus.cc:2758] T fccff64af8ca4400afc116db98918119 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
W20250112 02:52:34.454294 440 catalog_manager.cc:4945] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): alter failed for tablet c04b64bc2f8348a086ed3c4e59f09b16 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]),no further retry: Not found: Tablet not found: c04b64bc2f8348a086ed3c4e59f09b16
I20250112 02:52:34.455492 2024 ts_tablet_manager.cc:1918] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:34.455698 2024 log.cc:1198] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/d659c5a4f7a446c6aaf2283f65b45106
I20250112 02:52:34.456270 2024 ts_tablet_manager.cc:1939] T d659c5a4f7a446c6aaf2283f65b45106 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:34.458564 2024 tablet_replica.cc:331] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
W20250112 02:52:34.460595 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f16ec377f01484eae10e63490da143f has no consensus state
I20250112 02:52:34.461004 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_977c68d65cd443ef936f3dd77b445154" } new_table_name: "default.Table_7cbe570a89784c18b4a14a8242062071" modify_external_catalogs: true
I20250112 02:52:34.462620 2024 raft_consensus.cc:2238] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:34.463570 2024 raft_consensus.cc:2267] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:34.464908 2024 ts_tablet_manager.cc:1905] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:34.466656 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_5adbd32e9a0c4736844af4c72b7787fd" } new_table_name: "default.Table_556e3fb3ccfb4128b7224a024ad819ac" modify_external_catalogs: true
I20250112 02:52:34.461972 989 leader_election.cc:304] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:34.458842 1173 ts_tablet_manager.cc:1428] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20250112 02:52:34.467219 1173 tablet_bootstrap.cc:492] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:34.469194 1173 tablet_bootstrap.cc:654] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:34.469445 2056 raft_consensus.cc:2798] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:34.470077 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet cc62dcbf67e84e36ae1fbe127ccd4879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.470402 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c686fdc8896463c81c9f7c5f53a3da2 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c03b5e612d64314b210c11686fdfa2a) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.470675 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d411f906b4d4eb68343bdcc493e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.470921 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b521522c4784d2ab4056268823f1099 with delete_type TABLET_DATA_DELETED (replaced by tablet ab2e31a65294407bb62b13e92898a7d2) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.471616 2056 raft_consensus.cc:695] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:34.472486 2056 consensus_queue.cc:237] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.403337 2036 raft_consensus.cc:2238] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:34.475463 1177 tablet_bootstrap.cc:492] T 89b70dce4d1d4e29997a18297ac0f82a P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
W20250112 02:52:34.475605 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet bd5b3c712c0f448ea9b14ab93dd731ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd5b3c712c0f448ea9b14ab93dd731ab already in progress: creating tablet
I20250112 02:52:34.475708 1177 ts_tablet_manager.cc:1397] T 89b70dce4d1d4e29997a18297ac0f82a P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.004s
W20250112 02:52:34.475912 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b1d9fae290ec4281839991f9d40b105c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1d9fae290ec4281839991f9d40b105c already in progress: creating tablet
W20250112 02:52:34.476109 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 70715db0394d46d9b6b289160b00be7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70715db0394d46d9b6b289160b00be7d already in progress: creating tablet
I20250112 02:52:34.476294 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet d659c5a4f7a446c6aaf2283f65b45106 (table default.Table_218fd160176b4f55937d334142f2ecbd [id=990e8109058d4dffbbd0ae23fc43783b]) successfully deleted
I20250112 02:52:34.476845 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 49bb6d20cfa0453e90e048825114525e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.477857 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet f6803d0f553b4204b415b2734286bf4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.478579 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a3486ef011492c8a1df8d6278adedf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.479110 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 97b31ebe035c4552baaffc3133ec73c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.477443 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a7281c0c09442ebd0cc13b8496c86e" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:34.479669 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 64707ee432fb4e96a3bbf7252dfa1813 with delete_type TABLET_DATA_DELETED (replaced by tablet b3c5a861a90840d0a33b780ec75bffd2) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.480610 2036 raft_consensus.cc:2267] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:34.481244 2036 ts_tablet_manager.cc:1905] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:34.485772 854 leader_election.cc:343] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.486199 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5b3df88a0c45dba879b3d506710614" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:34.486377 916 raft_consensus.cc:2463] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
W20250112 02:52:34.491364 855 leader_election.cc:343] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.491501 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd498ccce484a89855c982762086890 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.491508 855 leader_election.cc:304] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:34.491727 2032 raft_consensus.cc:2743] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:34.491884 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 015412a5fa304453bd8d1be1c3cd7a46 with delete_type TABLET_DATA_DELETED (replaced by tablet f924459d1150472aafb325a11825db23) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:34.492079 855 leader_election.cc:343] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.492168 855 leader_election.cc:304] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:34.492195 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet f7cd0927c6a4401cbe69f55d4db9963a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.492473 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet aba1744138fe42ae9d097db01cac083d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.492482 2032 raft_consensus.cc:2743] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:34.492834 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 61593969d2234ce3a6089027b067d922 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:34.492854 855 leader_election.cc:343] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.493063 855 leader_election.cc:304] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:34.493252 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f22a7b1b804661af3b7e785a833739 with delete_type TABLET_DATA_DELETED (replaced by tablet 3d662e0ce13f447595e5f92b94de3018) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.493325 2032 raft_consensus.cc:2743] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:34.493687 855 leader_election.cc:343] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.493834 855 leader_election.cc:304] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:34.494267 2032 raft_consensus.cc:2743] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:34.421738 1181 tablet_bootstrap.cc:654] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:34.495949 2003 tablet_replica.cc:331] T aadb2ab1d2024f889841ac91e065e064 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:34.497535 918 raft_consensus.cc:2463] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:34.505512 2032 raft_consensus.cc:491] T e2f1f1a76e5f4308bf56dd647f02c6d8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.505650 2032 raft_consensus.cc:513] T e2f1f1a76e5f4308bf56dd647f02c6d8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:34.506295 2032 leader_election.cc:290] T e2f1f1a76e5f4308bf56dd647f02c6d8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:34.507118 453 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:38361 (request call id 209) took 1447 ms (client timeout 1000 ms). Trace:
W20250112 02:52:34.507261 453 rpcz_store.cc:269] 0112 02:52:33.060136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:33.060195 (+ 59us) service_pool.cc:225] Handling call
0112 02:52:34.507057 (+1446862us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:33.873836 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:33.873906 (+ 70us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:33.873911 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:33.873913 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:33.873983 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:33.873993 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:33.873999 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:33.874044 (+ 45us) tablet.cc:698] Row locks acquired
0112 02:52:33.874056 (+ 12us) write_op.cc:260] PREPARE: finished
0112 02:52:33.874080 (+ 24us) write_op.cc:270] Start()
0112 02:52:33.874101 (+ 21us) write_op.cc:276] Timestamp: P: 1736650353873981 usec, L: 0
0112 02:52:33.874104 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:33.874220 (+ 116us) log.cc:844] Serialized 527 byte log entry
0112 02:52:33.882452 (+ 8232us) op_driver.cc:464] REPLICATION: finished
0112 02:52:33.882574 (+ 122us) write_op.cc:301] APPLY: starting
0112 02:52:33.882586 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:33.882635 (+ 49us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:33.882663 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:33.882702 (+ 39us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:33.882728 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:33.882735 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:33.882784 (+ 49us) log.cc:844] Serialized 23 byte log entry
0112 02:52:33.882828 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:33.882840 (+ 12us) write_op.cc:454] Released schema lock
0112 02:52:33.882845 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:33.882906 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:33.882910 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6777,"reported_tablets":180,"child_traces":[["op",{"apply.queue_time_us":87,"apply.run_cpu_time_us":324,"apply.run_wall_time_us":323,"num_ops":1,"prepare.queue_time_us":347,"prepare.run_cpu_time_us":454,"prepare.run_wall_time_us":454,"raft.queue_time_us":803,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":8324}]]}
I20250112 02:52:34.509227 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet b7bd96fb381f4e73a0081be3fc799c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.510752 1173 tablet_bootstrap.cc:492] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:34.510970 1173 ts_tablet_manager.cc:1397] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.000s
I20250112 02:52:34.511862 1173 raft_consensus.cc:357] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.512243 1173 raft_consensus.cc:383] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:34.509873 1177 raft_consensus.cc:357] T 89b70dce4d1d4e29997a18297ac0f82a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.512523 1177 raft_consensus.cc:383] T 89b70dce4d1d4e29997a18297ac0f82a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:34.509119 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1e3af2883341e9b957f6ed748e604d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:34.512791 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a3486ef011492c8a1df8d6278adedf" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:34.513672 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5b3df88a0c45dba879b3d506710614" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:34.517043 855 leader_election.cc:343] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.517199 855 leader_election.cc:304] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:34.517625 2032 raft_consensus.cc:2743] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:34.518138 754 tablet_service.cc:1514] Processing DeleteTablet for tablet b61f16f3d601436bb4b32344de28c1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.518784 1181 tablet_bootstrap.cc:492] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:34.513907 751 tablet_service.cc:1514] Processing DeleteTablet for tablet bab4978edea0445fb0456e88e7fb4c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.518920 1181 ts_tablet_manager.cc:1397] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.119s user 0.001s sys 0.003s
I20250112 02:52:34.519625 776 raft_consensus.cc:3054] T 1a1e3af2883341e9b957f6ed748e604d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:34.519706 1181 raft_consensus.cc:357] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.519898 1181 raft_consensus.cc:383] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:34.519966 1181 raft_consensus.cc:738] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:34.521487 1181 consensus_queue.cc:260] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.522122 1181 ts_tablet_manager.cc:1428] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:34.522382 1181 tablet_bootstrap.cc:492] T 763f77385e404c149c16edabe469c299 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:34.522624 776 raft_consensus.cc:2463] T 1a1e3af2883341e9b957f6ed748e604d P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:34.523455 771 raft_consensus.cc:2463] T 9d5b3df88a0c45dba879b3d506710614 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:34.523723 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2f1f1a76e5f4308bf56dd647f02c6d8" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:34.525233 855 leader_election.cc:343] T e2f1f1a76e5f4308bf56dd647f02c6d8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.526033 1181 tablet_bootstrap.cc:654] T 763f77385e404c149c16edabe469c299 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:34.526218 2032 raft_consensus.cc:491] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.526342 2032 raft_consensus.cc:513] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.526965 2032 leader_election.cc:290] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
W20250112 02:52:34.527662 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 128909us
I20250112 02:52:34.536252 2071 tablet.cc:1721] T c69587c5a09d40f0baba5c2f6e8a7b41 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:34.536451 2036 ts_tablet_manager.cc:1918] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:34.536557 2036 log.cc:1198] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/f234256b343f447dadfc3d1936a824ee
I20250112 02:52:34.536883 2036 ts_tablet_manager.cc:1939] T f234256b343f447dadfc3d1936a824ee P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:34.513139 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet afa2f8aabe5e42ccbc15b193fdc03563 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.514433 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 1726d61cc36a4deb83c5d604e8daf4f0 with delete_type TABLET_DATA_DELETED (replaced by tablet b2ccb88108f24e36ba4b7ebf36398950) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.539628 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d84027ad6d40a5be5afcdd854431a5 with delete_type TABLET_DATA_DELETED (replaced by tablet fbe9e9624ede40c29ea852b2b5494965) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.515789 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet e9fc9a4f1e8944abb76dad6c2ad53d35 with delete_type TABLET_DATA_DELETED (replaced by tablet 5cc414d6420c41839621ab858067548b) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.539877 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 04f135b226774285a077e5806131d91b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:34.515971 991 leader_election.cc:343] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:34.540184 991 leader_election.cc:343] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.516953 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 87759e8b99b04e628badc8f60cb38bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.519184 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c639a2bcd4740e4be7b2b162004669a with delete_type TABLET_DATA_DELETED (replaced by tablet 5d0c3191973a477e949fbf9dfcca963f) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.519877 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet fb0aba0139cf479b80b32f3dfc936535 with delete_type TABLET_DATA_DELETED (replaced by tablet e2f1f1a76e5f4308bf56dd647f02c6d8) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.520242 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 416fb6820cd54e118950c68d381f51b2) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.541054 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a18132549374218893f35a4e4226b72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.541114 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet a85300a9aa1c4c50a3957a004bea7933 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.541278 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb76746b3a846c3aba531c04bf63966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.541299 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 73136be25c79491784d3ea71aa9b27f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.541419 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet d45fce86e02f46d4a57af718fa2a62c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 5804fc4130f84c69832c1022b7bd7fa5) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.541466 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 2730634a5cfd4c2fb5c4e30100c556d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.541563 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet a12c5d58e6c246c983e8a533906ce77d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.541663 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet b61f16f3d601436bb4b32344de28c1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.541707 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet b9c006281dc949119951c74e23167d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.541813 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b184806b4884cd3b0a002905a1e57af with delete_type TABLET_DATA_DELETED (replaced by tablet 47a0d939fdb94590a86389f1f4693e9a) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.541923 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet f234256b343f447dadfc3d1936a824ee with delete_type TABLET_DATA_DELETED (replaced by tablet 82b351c563d749df806ab5d0331a4ef2) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.541967 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 06edfca60eed4dd3a2dee7b662b0970c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.542140 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.542205 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a7281c0c09442ebd0cc13b8496c86e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.541026 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet b2ccb88108f24e36ba4b7ebf36398950 with delete_type TABLET_DATA_DELETED (replaced by tablet f3771166b77e4f90aaa116ab1a9dc299) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.542328 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c821ea3ca4946c290c17135366e0cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.542382 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.520422 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d5b3df88a0c45dba879b3d506710614 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.520491 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a8a178bafcb457f9b0e9587df560493 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.520573 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a34d03a80ae402db6607261a5c40b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.521903 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet ea33e5c1a1fb46599c5836a36b130051 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.532452 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet bb217afc08304c0ea35dee3120347ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.532802 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c639a2bcd4740e4be7b2b162004669a) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.532884 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a12b225131b464494a9f02133d37aea with delete_type TABLET_DATA_DELETED (replaced by tablet d45fce86e02f46d4a57af718fa2a62c8) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.533557 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet f05c8426ea3d4fe89fbe6c88c688019f with delete_type TABLET_DATA_DELETED (replaced by tablet c5079f87652a4ea58b366d1f9448fd4e) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.533865 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet daef3a58605e40978f7389698b22265d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.533986 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 9163878cfeb64693a9c5493e812546af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.534346 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd48d0e68014e1aa0c44620527baec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.542891 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet d4abf59d96414e5faa7cf75ba133832c with delete_type TABLET_DATA_DELETED (replaced by tablet 5a3af0ea4e954053ae89814f3e74e6b7) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.534631 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet e5c10fad790046a88fbc8ebe0303c481 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.534739 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b599e983ec04f53a440d1ab828d12ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.542158 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 352c87f1dc724e7b94bc2e64abfce8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.542479 1173 raft_consensus.cc:738] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:34.543824 1173 consensus_queue.cc:260] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.548712 2024 ts_tablet_manager.cc:1918] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:34.548926 2024 log.cc:1198] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/02d0f67141dc46b6a87ca0ad080d5bfe
I20250112 02:52:34.549479 2024 ts_tablet_manager.cc:1939] T 02d0f67141dc46b6a87ca0ad080d5bfe P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:34.555981 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 62f4bd2df0ce45dabc55373f1154d3ac
I20250112 02:52:34.556443 454 catalog_manager.cc:6596] ReplaceTablet: tablet 62f4bd2df0ce45dabc55373f1154d3ac deleted and replaced by tablet f559dedfb7854cf890bef362c50b4213
I20250112 02:52:34.556787 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_4603068c01544e2e965dca5177e36b40" } modify_external_catalogs: true
I20250112 02:52:34.556941 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_4603068c01544e2e965dca5177e36b40" } modify_external_catalogs: true
I20250112 02:52:34.560493 742 tablet_service.cc:1467] Processing CreateTablet for tablet 500f0c5c76e54cd8808e6c8611e99d83 (DEFAULT_TABLE table=default.Table_67e40ef8ee824430b5d090ddc9cc03c8 [id=5c61d80dfd2746f591c7296315935aaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.563282 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet a8e7882e07784bdda18dc1ffd20963c8
I20250112 02:52:34.563999 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_849b7f79a5784f5fabed0136deb0a129" } new_table_name: "default.Table_5e7392cb5a68412f97bddb2677986918" modify_external_catalogs: true
I20250112 02:52:34.566532 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 8ee98f055024429eafaa29bb961cab87
I20250112 02:52:34.566969 452 catalog_manager.cc:6596] ReplaceTablet: tablet 8ee98f055024429eafaa29bb961cab87 deleted and replaced by tablet 99407ead9cf547e3bbe78c0d0b327e6a
I20250112 02:52:34.567258 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_9284616ca0cd499cb51df0322db9948f" } modify_external_catalogs: true
I20250112 02:52:34.567416 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_9284616ca0cd499cb51df0322db9948f" } modify_external_catalogs: true
I20250112 02:52:34.548573 2071 tablet.cc:1721] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:34.555004 899 tablet_service.cc:1467] Processing CreateTablet for tablet 500f0c5c76e54cd8808e6c8611e99d83 (DEFAULT_TABLE table=default.Table_67e40ef8ee824430b5d090ddc9cc03c8 [id=5c61d80dfd2746f591c7296315935aaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.572273 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 500f0c5c76e54cd8808e6c8611e99d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.573750 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 65f22a7b1b804661af3b7e785a833739 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]) successfully deleted
W20250112 02:52:34.574417 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet bd5b3c712c0f448ea9b14ab93dd731ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd5b3c712c0f448ea9b14ab93dd731ab already in progress: creating tablet
I20250112 02:52:34.558821 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f4bd2df0ce45dabc55373f1154d3ac with delete_type TABLET_DATA_DELETED (replaced by tablet f559dedfb7854cf890bef362c50b4213) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.563066 898 tablet_service.cc:1467] Processing CreateTablet for tablet 0f16ec377f01484eae10e63490da143f (DEFAULT_TABLE table=default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.576012 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f16ec377f01484eae10e63490da143f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.531618 1177 raft_consensus.cc:738] T 89b70dce4d1d4e29997a18297ac0f82a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:34.563935 892 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e7882e07784bdda18dc1ffd20963c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.567298 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ee98f055024429eafaa29bb961cab87 with delete_type TABLET_DATA_DELETED (replaced by tablet 99407ead9cf547e3bbe78c0d0b327e6a) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.571594 890 tablet_service.cc:1467] Processing CreateTablet for tablet 09900b42e8a443e680cc35c96a1dbbb7 (DEFAULT_TABLE table=default.Table_5e1a058646834ea99def3d8262bee81b [id=ec894df00ecb4b62881cbc11cdede0a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.580049 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09900b42e8a443e680cc35c96a1dbbb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.581719 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 06edfca60eed4dd3a2dee7b662b0970c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.581931 754 tablet_service.cc:1514] Processing DeleteTablet for tablet b9c006281dc949119951c74e23167d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.582110 744 tablet_service.cc:1514] Processing DeleteTablet for tablet f234256b343f447dadfc3d1936a824ee with delete_type TABLET_DATA_DELETED (replaced by tablet 82b351c563d749df806ab5d0331a4ef2) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.582226 2003 raft_consensus.cc:2238] T aadb2ab1d2024f889841ac91e065e064 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:34.582409 2003 raft_consensus.cc:2267] T aadb2ab1d2024f889841ac91e065e064 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:34.583127 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c821ea3ca4946c290c17135366e0cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.583317 755 tablet_service.cc:1514] Processing DeleteTablet for tablet b1d9fae290ec4281839991f9d40b105c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.583496 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.583544 759 tablet_service.cc:1514] Processing DeleteTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.583667 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 500f0c5c76e54cd8808e6c8611e99d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.586974 2071 tablet.cc:1721] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:34.587574 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f4bd2df0ce45dabc55373f1154d3ac with delete_type TABLET_DATA_DELETED (replaced by tablet f559dedfb7854cf890bef362c50b4213) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.588016 2003 ts_tablet_manager.cc:1905] T aadb2ab1d2024f889841ac91e065e064 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:34.589103 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet c29a9ca43e034b43a00207e5847bbee3
I20250112 02:52:34.589828 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_b1a770633f004b01b66ddfa83e0fd698" } modify_external_catalogs: true
I20250112 02:52:34.591347 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_b1a770633f004b01b66ddfa83e0fd698" } modify_external_catalogs: true
I20250112 02:52:34.594806 2094 tablet.cc:1721] T c69587c5a09d40f0baba5c2f6e8a7b41 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:34.597448 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 058a9ca089cd4967887b0a1127c7d03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.598193 1177 consensus_queue.cc:260] T 89b70dce4d1d4e29997a18297ac0f82a P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.599082 898 tablet_service.cc:1514] Processing DeleteTablet for tablet c29a9ca43e034b43a00207e5847bbee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.601032 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2361bc6bd8164c2daa37be6b0845f4bd" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:34.603137 2071 tablet.cc:1721] T e3ee5d8a17874d7ba54de51568e7d2e6 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:52:34.603847 855 leader_election.cc:343] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.604945 1177 ts_tablet_manager.cc:1428] T 89b70dce4d1d4e29997a18297ac0f82a P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.129s user 0.003s sys 0.000s
I20250112 02:52:34.605299 1177 tablet_bootstrap.cc:492] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:34.607414 714 leader_election.cc:343] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 5dd48d0e68014e1aa0c44620527baec9
W20250112 02:52:34.609566 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 69023us
W20250112 02:52:34.609853 991 consensus_peers.cc:487] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:34.609859 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 83c04f1a2cb04950a9f6170e252c19b4
I20250112 02:52:34.610288 991 leader_election.cc:304] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:34.610551 991 leader_election.cc:304] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:34.610939 2094 tablet.cc:1721] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:34.610978 2052 consensus_queue.cc:1035] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:34.611233 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c04f1a2cb04950a9f6170e252c19b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:34.611384 2052 consensus_queue.cc:1035] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:34.611749 2052 raft_consensus.cc:2798] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:34.611894 2052 raft_consensus.cc:695] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:34.612133 2052 consensus_queue.cc:237] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.612957 1727 raft_consensus.cc:2798] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:34.613062 1727 raft_consensus.cc:491] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:34.613134 1727 raft_consensus.cc:3054] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:34.617671 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet f234256b343f447dadfc3d1936a824ee (table default.Table_4603068c01544e2e965dca5177e36b40 [id=8502f21e9b61436093db1241cfbb5ec9]) successfully deleted
I20250112 02:52:34.622920 2160 tablet.cc:1721] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:34.623347 714 leader_election.cc:304] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:34.627393 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_36b06d2edb8e4d23a3cb7021a8267db4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:34.628068 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36b06d2edb8e4d23a3cb7021a8267db4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:34.629210 2094 tablet.cc:1721] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:34.632133 898 tablet_service.cc:1467] Processing CreateTablet for tablet 83c04f1a2cb04950a9f6170e252c19b4 (DEFAULT_TABLE table=default.Table_b1a770633f004b01b66ddfa83e0fd698 [id=20fddd79df734f3bb4552ac2ca4958e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.632460 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83c04f1a2cb04950a9f6170e252c19b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.636133 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_c3e55b9696194ba58cbc9f1bb391b27d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:34.636682 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3e55b9696194ba58cbc9f1bb391b27d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:34.640398 1609 raft_consensus.cc:3054] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:34.644693 2036 tablet_replica.cc:331] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:34.644941 2036 raft_consensus.cc:2238] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:34.649848 2158 tablet.cc:1721] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250112 02:52:34.650346 2036 raft_consensus.cc:2267] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:34.652647 2036 ts_tablet_manager.cc:1905] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:34.653923 1727 raft_consensus.cc:513] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.650118 898 tablet_service.cc:1467] Processing CreateTablet for tablet 3499374810fb412e938325434c35618d (DEFAULT_TABLE table=default.Table_7870d93893b74b4b96d1a3809a9015e6 [id=4355589b5ccc4162ab8700e31173d1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.658241 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3499374810fb412e938325434c35618d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.651360 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_84016330aad94c2ba3aec196a69d28de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:34.661412 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84016330aad94c2ba3aec196a69d28de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:34.664160 1727 leader_election.cc:290] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:34.669402 1173 ts_tablet_manager.cc:1428] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.144s user 0.002s sys 0.000s
I20250112 02:52:34.671289 2117 raft_consensus.cc:491] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.671509 2117 raft_consensus.cc:513] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.675647 2116 raft_consensus.cc:491] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.675786 2116 raft_consensus.cc:513] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.676402 2116 leader_election.cc:290] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:34.677031 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_70d1488de98d4380b8024db852f58991"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:34.677954 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2f1f1a76e5f4308bf56dd647f02c6d8" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:34.672842 2117 leader_election.cc:290] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
W20250112 02:52:34.684108 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70d1488de98d4380b8024db852f58991 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:34.684187 890 tablet_service.cc:1467] Processing CreateTablet for tablet 2d47f6d663934b25a0200ee9fa81165d (DEFAULT_TABLE table=default.Table_fbc52174b22b4aafa4ede40120b90daf [id=4418eb326c7342e6a8c16a263827adbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.684855 1181 tablet_bootstrap.cc:492] T 763f77385e404c149c16edabe469c299 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:34.684996 1181 ts_tablet_manager.cc:1397] T 763f77385e404c149c16edabe469c299 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.163s user 0.005s sys 0.003s
I20250112 02:52:34.685453 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d47f6d663934b25a0200ee9fa81165d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.687516 1181 raft_consensus.cc:357] T 763f77385e404c149c16edabe469c299 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.687686 1181 raft_consensus.cc:383] T 763f77385e404c149c16edabe469c299 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:34.687726 1181 raft_consensus.cc:738] T 763f77385e404c149c16edabe469c299 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:34.687891 1181 consensus_queue.cc:260] T 763f77385e404c149c16edabe469c299 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.692602 2036 ts_tablet_manager.cc:1918] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:34.692691 2036 log.cc:1198] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/c8a7281c0c09442ebd0cc13b8496c86e
I20250112 02:52:34.692996 2036 ts_tablet_manager.cc:1939] T c8a7281c0c09442ebd0cc13b8496c86e P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:34.686424 899 tablet_service.cc:1467] Processing CreateTablet for tablet 54d3664071b146a3ac7c6885f9bda789 (DEFAULT_TABLE table=default.Table_787c491d6a594a27b46814669b3ccc2b [id=e14369e145934ad9b3c515b33ad29a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.696091 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54d3664071b146a3ac7c6885f9bda789. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.700784 756 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e7882e07784bdda18dc1ffd20963c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.701202 1177 tablet_bootstrap.cc:654] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:34.701954 742 tablet_service.cc:1467] Processing CreateTablet for tablet 09900b42e8a443e680cc35c96a1dbbb7 (DEFAULT_TABLE table=default.Table_5e1a058646834ea99def3d8262bee81b [id=ec894df00ecb4b62881cbc11cdede0a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.703079 1181 ts_tablet_manager.cc:1428] T 763f77385e404c149c16edabe469c299 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20250112 02:52:34.703123 748 tablet_service.cc:1467] Processing CreateTablet for tablet 0f16ec377f01484eae10e63490da143f (DEFAULT_TABLE table=default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.703459 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795ac7eaf81d48bb926865d9ebb388b6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:34.704762 855 leader_election.cc:343] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.705338 1181 tablet_bootstrap.cc:492] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:34.705492 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ee98f055024429eafaa29bb961cab87 with delete_type TABLET_DATA_DELETED (replaced by tablet 99407ead9cf547e3bbe78c0d0b327e6a) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.706248 2117 raft_consensus.cc:491] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.706377 2117 raft_consensus.cc:513] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.706838 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09900b42e8a443e680cc35c96a1dbbb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.707656 2117 leader_election.cc:290] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:34.709983 756 tablet_service.cc:1514] Processing DeleteTablet for tablet c29a9ca43e034b43a00207e5847bbee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.711441 1181 tablet_bootstrap.cc:654] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:34.711645 899 tablet_service.cc:1467] Processing CreateTablet for tablet d95f98ae7bdf42518bbff55e257fea88 (DEFAULT_TABLE table=default.Table_dd77b8845fdd46b7a2debbadaf553f75 [id=70c61753e9944e1fbd65e9ec41a48ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.712136 2117 raft_consensus.cc:491] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.712263 2117 raft_consensus.cc:513] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:34.713341 2117 leader_election.cc:290] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:34.714915 895 tablet_service.cc:1467] Processing CreateTablet for tablet 3efcd32b4ccc4fada5adb3ca3bcdddaa (DEFAULT_TABLE table=default.Table_28b2d14acaa04d2a90164800ca8ced3c [id=afa0a72f5c634b289342dc0872945868]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:34.718056 2176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:52:34.679919 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
W20250112 02:52:34.718585 2174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:52:34.719077 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5cc414d6420c41839621ab858067548b (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]). Expected version 1 got 0
I20250112 02:52:34.719334 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
I20250112 02:52:34.719695 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 7ea075826b16475ca474e284fc6bc545 (table default.Table_009a6976f6db4f218e3c3373368a58d7 [id=c5005669bb5f4c029d673bd39cc1535d]). Expected version 1 got 0
I20250112 02:52:34.720054 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:34.720296 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet e3ee5d8a17874d7ba54de51568e7d2e6 (table default.Table_83177e7dc14645dbb0e5ef573e80e72f [id=2c077da077834ae4bcc84efdd9e89120]). Expected version 1 got 0
I20250112 02:52:34.720489 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:34.720655 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet fccff64af8ca4400afc116db98918119 (table default.Table_15bcb2c0f2a74458a1bed231fd3437fc [id=ca9c8281bf254f248076cb600e7c4f87]). Expected version 2 got 0
I20250112 02:52:34.720655 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet c8a7281c0c09442ebd0cc13b8496c86e (table default.Table_105e5d1a8ef846ffbcb1bdbe01bbae55 [id=082744a588e64fe8bafe0bc442876fc9]) successfully deleted
I20250112 02:52:34.720978 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet f924459d1150472aafb325a11825db23 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]). Expected version 3 got 0
I20250112 02:52:34.721222 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:34.721731 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:34.722991 2036 tablet_replica.cc:331] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:34.723076 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3efcd32b4ccc4fada5adb3ca3bcdddaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.723510 2036 raft_consensus.cc:2238] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:34.724433 2036 raft_consensus.cc:2267] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:34.728235 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d95f98ae7bdf42518bbff55e257fea88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.728466 2036 ts_tablet_manager.cc:1905] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:34.735060 1173 tablet_bootstrap.cc:492] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:34.739859 2170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:52:34.741461 892 tablet_service.cc:1467] Processing CreateTablet for tablet 61eebc5f22664712af6e74e82809ca87 (DEFAULT_TABLE table=default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.742201 887 tablet_service.cc:1467] Processing CreateTablet for tablet f3771166b77e4f90aaa116ab1a9dc299 (DEFAULT_TABLE table=default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.746201 890 tablet_service.cc:1467] Processing CreateTablet for tablet 781a9943f705477cb6eae16df4468d2f (DEFAULT_TABLE table=default.Table_104ccd5f390f47a5878c67ea6de9d554 [id=872482df2ee94056a7d29ce227797f16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.747431 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61eebc5f22664712af6e74e82809ca87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.748430 889 tablet_service.cc:1467] Processing CreateTablet for tablet ea33053c62dc4707b759d5c1756ab4bd (DEFAULT_TABLE table=default.Table_489753b0c1154a4ba82ad30de84de181 [id=1b3a8812e2384219b500040a72efc1b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.721985 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet abfaad67d5e34c0a8338dccc0ef05466 (table default.Table_045e5b104dc64217aa39c22534007cab [id=2c0e94e8c5774da1b8130c7651d94500]). Expected version 1 got 0
I20250112 02:52:34.749029 891 tablet_service.cc:1467] Processing CreateTablet for tablet a8e7882e07784bdda18dc1ffd20963c8 (DEFAULT_TABLE table=default.Table_f3695b00a89b4353903587a6c1418d8c [id=d1781738b8a74f139b09863918cbb21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.749282 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 09d06e2e8c13464db8cf801a292bdc99 (table default.Table_0bdc5f49d4ff4d6fb1290bc2dc0f70d9 [id=e3bb489c53454222a79702b798c0c2f1]). Expected version 2 got 0
I20250112 02:52:34.749339 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 781a9943f705477cb6eae16df4468d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.749634 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8e7882e07784bdda18dc1ffd20963c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.749764 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:34.749969 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c69587c5a09d40f0baba5c2f6e8a7b41 (table default.Table_1216acc1fa6b4e51a55a89ffe22bcc9a [id=b5457e4343ee43618c8309e737b69ec3]). Expected version 1 got 0
I20250112 02:52:34.750120 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:34.750955 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_5d473cabff2e46e0b531fbce28d9d558 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 1 got 0
I20250112 02:52:34.751108 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:34.751251 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:34.751554 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet d8eb0efc5d7f4c72bdfc7531a444d42a (table default.Table_d561e1740cc545dc8aa4babb76b45c37 [id=de27aeff6b3a49b3956498ea5015ff33]). Expected version 1 got 0
I20250112 02:52:34.751845 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 4916ae3917084cfe8c93240a8857be0e (table default.Table_69f363c4da234792886b544823b3214e [id=c7052afaf6284ef0972835c1f46cb829]). Expected version 1 got 0
I20250112 02:52:34.752014 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:34.752316 2028 tablet.cc:1721] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:34.752380 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 23e7a2e88c374e949ced92c557225cbf (table default.Table_cd1f185d82b3474083246ec3702851fc [id=bfa9a504a8d34c86929efb0d15d049d0]). Expected version 3 got 0
I20250112 02:52:34.752808 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 2c03b5e612d64314b210c11686fdfa2a (table default.Table_4934f6022f2441eda88751236a5798be [id=5c4769bfcefd4fe0a0f3c2e315d50c32]). Expected version 6 got 0
I20250112 02:52:34.753059 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:34.753716 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet f8c01b80fc8242d98fdfbf767212144d (table default.Table_63649e9607394f4bb72a3dd6e6d0dd69 [id=9807191b6e5c4d478cd47a54464fb9ce]). Expected version 1 got 0
I20250112 02:52:34.753958 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 6117dc8b32a64535a2f4167a03bed0d2 (table default.Table_365739770a11434dbf260ea912e7ea81 [id=868adb627654450c936ef198e49d853b]). Expected version 1 got 0
I20250112 02:52:34.754081 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2361bc6bd8164c2daa37be6b0845f4bd" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:34.754278 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:34.754595 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet f31b777e7e0f43a28839d8c48c6a6590 (table default.Table_5fb592e41cc4406aa27546e7b41fdb88 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 1 got 0
I20250112 02:52:34.754839 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:34.755127 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:34.755561 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea33053c62dc4707b759d5c1756ab4bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.755846 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3771166b77e4f90aaa116ab1a9dc299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.755800 1727 raft_consensus.cc:491] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.755965 1727 raft_consensus.cc:513] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.756768 1727 leader_election.cc:290] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:34.760594 2117 raft_consensus.cc:491] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.763875 2028 tablet.cc:1721] T 7cd9985cd20148f195a857c478b0755f P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:34.765316 2117 raft_consensus.cc:513] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.774135 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856fa83dedfd4b049e3e0d91d7ac5e40" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:34.774808 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_b236e732be7844a08eaa268cb451d22f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:34.774909 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05c8426ea3d4fe89fbe6c88c688019f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:34.775369 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b236e732be7844a08eaa268cb451d22f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:34.775728 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45fce86e02f46d4a57af718fa2a62c8" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:34.776335 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8daf4ef3593410eae87672402292be6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:34.776859 989 leader_election.cc:304] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:34.777812 2050 raft_consensus.cc:2798] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:34.777969 2050 raft_consensus.cc:695] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:34.778219 2050 consensus_queue.cc:237] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.779335 1727 consensus_queue.cc:1035] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:52:34.784015 2116 raft_consensus.cc:491] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.784236 2116 raft_consensus.cc:513] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.785022 2116 leader_election.cc:290] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:34.785044 2117 leader_election.cc:290] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:34.786041 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_cc762fcab6cf493ab58b3d1defecf106"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:34.786967 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc762fcab6cf493ab58b3d1defecf106 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:34.787922 1173 tablet_bootstrap.cc:654] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:34.799494 898 tablet_service.cc:1467] Processing CreateTablet for tablet 3d418cda9ed5449fa45690130cd6d5bc (DEFAULT_TABLE table=default.Table_8f2b912f93d84320b0f7e127c0543af6 [id=a49da46bf16343a488f96156139990a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.799942 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d418cda9ed5449fa45690130cd6d5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.800410 2036 ts_tablet_manager.cc:1918] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:34.800573 2036 log.cc:1198] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/a12c5d58e6c246c983e8a533906ce77d
I20250112 02:52:34.801026 2036 ts_tablet_manager.cc:1939] T a12c5d58e6c246c983e8a533906ce77d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:34.801330 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795ac7eaf81d48bb926865d9ebb388b6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:34.802027 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet a12c5d58e6c246c983e8a533906ce77d (table default.Table_f7bab317410d408aa363857a99be6e3c [id=d0be0fa53a79498e9160bfaa7d88837b]) successfully deleted
I20250112 02:52:34.802453 2036 tablet_replica.cc:331] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:34.802675 2036 raft_consensus.cc:2238] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:34.802858 2036 raft_consensus.cc:2267] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:34.803401 2036 ts_tablet_manager.cc:1905] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:34.804051 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5b3df88a0c45dba879b3d506710614" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:34.804220 914 raft_consensus.cc:3054] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:34.805241 854 leader_election.cc:343] T e2f1f1a76e5f4308bf56dd647f02c6d8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.805352 854 leader_election.cc:304] T e2f1f1a76e5f4308bf56dd647f02c6d8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:34.806524 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 02d0f67141dc46b6a87ca0ad080d5bfe (table default.Table_3a84890c771c4574a25d33543fe7217d [id=49a61d0f688b488da9a139f0a64d2834]) successfully deleted
I20250112 02:52:34.806511 899 tablet_service.cc:1467] Processing CreateTablet for tablet a470f84d50df4cf0b4a1e5d248b4c32a (DEFAULT_TABLE table=default.Table_6e77252e817f43399c9b3f5b221795ad [id=1fa1474ab8804667b22e5f30c48c1630]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.806744 914 raft_consensus.cc:2463] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:34.806809 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a470f84d50df4cf0b4a1e5d248b4c32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.806886 2117 raft_consensus.cc:2743] T e2f1f1a76e5f4308bf56dd647f02c6d8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:34.815121 452 master_service.cc:946] ReplaceTablet: received request to replace tablet daef3a58605e40978f7389698b22265d from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:34.816548 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_ef2001a2d9b24b4bb1f8fb9102703022"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:34.817975 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef2001a2d9b24b4bb1f8fb9102703022 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:34.825305 2036 ts_tablet_manager.cc:1918] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:34.825810 2036 log.cc:1198] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/6cb76746b3a846c3aba531c04bf63966
I20250112 02:52:34.826339 2036 ts_tablet_manager.cc:1939] T 6cb76746b3a846c3aba531c04bf63966 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
W20250112 02:52:34.826659 854 leader_election.cc:343] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.827190 892 tablet_service.cc:1467] Processing CreateTablet for tablet 5450777104da48f1821575658fd31afd (DEFAULT_TABLE table=default.Table_af9d530db1774d9eafed6727be07aeaf [id=14b00b3f93ef44269cf7de795ab41492]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.827580 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5450777104da48f1821575658fd31afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.828317 889 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecf730b6985404a8134b497a4b792c8 (DEFAULT_TABLE table=default.Table_24fca5290fd54774bee413c9d84a997d [id=b9950eabab874da9a52a99555dfbefd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.828603 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ecf730b6985404a8134b497a4b792c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.829509 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 500f0c5c76e54cd8808e6c8611e99d83 (DEFAULT_TABLE table=default.Table_67e40ef8ee824430b5d090ddc9cc03c8 [id=5c61d80dfd2746f591c7296315935aaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.829952 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 500f0c5c76e54cd8808e6c8611e99d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.833160 854 leader_election.cc:304] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:34.833796 854 leader_election.cc:343] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:34.834174 854 leader_election.cc:343] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:34.834380 854 leader_election.cc:343] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:34.834597 854 leader_election.cc:343] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.836032 889 tablet_service.cc:1467] Processing CreateTablet for tablet 17c54c5dc86446cf85974777c8145ef6 (DEFAULT_TABLE table=default.Table_ab4d7c58924a4b8682b52637c23f125b [id=a622f7a973344a388b3275adbcce6234]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.837188 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17c54c5dc86446cf85974777c8145ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.838671 2117 raft_consensus.cc:2743] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:34.838848 899 tablet_service.cc:1467] Processing CreateTablet for tablet 42eed0a9f8354e5391ec2463dafdb683 (DEFAULT_TABLE table=default.Table_a5f330af613f4e5e9ecafdfe58eb0f1b [id=839e694f8f4f4424835ed91f8c604735]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.839231 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42eed0a9f8354e5391ec2463dafdb683. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.839988 2024 tablet_replica.cc:331] T 161fe26b9c0e4543b7aeb0021506d72a P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:34.840255 2024 raft_consensus.cc:2238] T 161fe26b9c0e4543b7aeb0021506d72a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:34.840541 2024 raft_consensus.cc:2267] T 161fe26b9c0e4543b7aeb0021506d72a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:34.841270 2024 ts_tablet_manager.cc:1905] T 161fe26b9c0e4543b7aeb0021506d72a P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:34.844873 892 tablet_service.cc:1467] Processing CreateTablet for tablet fc52271535df47e282bf1625c2e8e041 (DEFAULT_TABLE table=default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.845264 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc52271535df47e282bf1625c2e8e041. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.755414 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 003a3b4705e943c6a900762ec11ddd47 (table default.Table_743b3f7029534564b35c0c137b35fd75 [id=b2fa3922f6d447769a8179052f9a1861]). Expected version 1 got 0
I20250112 02:52:34.845878 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 7cd9985cd20148f195a857c478b0755f (table default.Table_9b65dd291c9f4fcdaa6d5e37f04d24ae [id=e63daf724d874c89badafb8d59653f74]). Expected version 1 got 0
I20250112 02:52:34.846134 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:34.846316 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
I20250112 02:52:34.846455 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:34.846738 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:34.847115 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:34.851595 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 6cb76746b3a846c3aba531c04bf63966 (table default.Table_75db5f75b2874392b44dd580e0e52de9 [id=013bd2f66fe64ed593d8c0261ffa1c18]) successfully deleted
I20250112 02:52:34.855916 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f16ec377f01484eae10e63490da143f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.857880 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e7882e07784bdda18dc1ffd20963c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.858621 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 0f16ec377f01484eae10e63490da143f (DEFAULT_TABLE table=default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.859038 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f16ec377f01484eae10e63490da143f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.859714 2036 tablet_replica.cc:331] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:34.859910 2036 raft_consensus.cc:2238] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:34.860080 2036 raft_consensus.cc:2267] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:34.860566 2036 ts_tablet_manager.cc:1905] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:34.866873 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet c29a9ca43e034b43a00207e5847bbee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.867105 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c04f1a2cb04950a9f6170e252c19b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.867424 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 09900b42e8a443e680cc35c96a1dbbb7 (DEFAULT_TABLE table=default.Table_5e1a058646834ea99def3d8262bee81b [id=ec894df00ecb4b62881cbc11cdede0a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.867877 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09900b42e8a443e680cc35c96a1dbbb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.876135 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f4bd2df0ce45dabc55373f1154d3ac with delete_type TABLET_DATA_DELETED (replaced by tablet f559dedfb7854cf890bef362c50b4213) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.876482 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ee98f055024429eafaa29bb961cab87 with delete_type TABLET_DATA_DELETED (replaced by tablet 99407ead9cf547e3bbe78c0d0b327e6a) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:34.871501 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856fa83dedfd4b049e3e0d91d7ac5e40" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:34.883818 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c04f1a2cb04950a9f6170e252c19b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:34.886535 2094 tablet.cc:1721] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250112 02:52:34.887143 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45fce86e02f46d4a57af718fa2a62c8" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:34.887763 1609 raft_consensus.cc:2743] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:34.889148 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05c8426ea3d4fe89fbe6c88c688019f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:34.896075 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_b30bfb8db92545468698dde44b7c4200"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:34.896678 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b30bfb8db92545468698dde44b7c4200 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:34.900477 899 tablet_service.cc:1467] Processing CreateTablet for tablet 76009bac22ce4136b148b493a004a0f7 (DEFAULT_TABLE table=default.Table_ea881fde8d694f71b5faf60948630d96 [id=ce171e0d83bd436eaad276bedf32030e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.900862 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76009bac22ce4136b148b493a004a0f7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:34.902240 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 289502us
I20250112 02:52:34.905118 2167 raft_consensus.cc:491] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.905303 2167 raft_consensus.cc:513] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.906098 2167 leader_election.cc:290] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:34.907613 890 tablet_service.cc:1467] Processing CreateTablet for tablet dcb7e0181cf74bbea6036c106516a7b3 (DEFAULT_TABLE table=default.Table_9d8f183a5347440f898ef33ee7299613 [id=c2e1172634dd4463b2a809de96e0450e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.907938 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcb7e0181cf74bbea6036c106516a7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.911402 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 3efcd32b4ccc4fada5adb3ca3bcdddaa (DEFAULT_TABLE table=default.Table_28b2d14acaa04d2a90164800ca8ced3c [id=afa0a72f5c634b289342dc0872945868]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.911513 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 54d3664071b146a3ac7c6885f9bda789 (DEFAULT_TABLE table=default.Table_787c491d6a594a27b46814669b3ccc2b [id=e14369e145934ad9b3c515b33ad29a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.911856 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3efcd32b4ccc4fada5adb3ca3bcdddaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.912030 1022 tablet_service.cc:1467] Processing CreateTablet for tablet d95f98ae7bdf42518bbff55e257fea88 (DEFAULT_TABLE table=default.Table_dd77b8845fdd46b7a2debbadaf553f75 [id=70c61753e9944e1fbd65e9ec41a48ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.912384 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d95f98ae7bdf42518bbff55e257fea88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.915705 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54d3664071b146a3ac7c6885f9bda789. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.915884 889 tablet_service.cc:1467] Processing CreateTablet for tablet 93aba9b4d1dd485cb62c4827c916a1d5 (DEFAULT_TABLE table=default.Table_d5f4d4c431d24cc0ba82a7dcd012368b [id=3059ab365aff4430b1570224188486fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.916191 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93aba9b4d1dd485cb62c4827c916a1d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.920639 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 2d47f6d663934b25a0200ee9fa81165d (DEFAULT_TABLE table=default.Table_fbc52174b22b4aafa4ede40120b90daf [id=4418eb326c7342e6a8c16a263827adbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.921216 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d47f6d663934b25a0200ee9fa81165d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.922188 2036 ts_tablet_manager.cc:1918] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:34.922288 2036 log.cc:1198] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/73136be25c79491784d3ea71aa9b27f2
I20250112 02:52:34.922581 2036 ts_tablet_manager.cc:1939] T 73136be25c79491784d3ea71aa9b27f2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:34.923624 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 73136be25c79491784d3ea71aa9b27f2 (table default.Table_e3c5ba009c9f4301a3f99d51a8dcf687 [id=d5bf19dc7bff469ba06018568b796a63]) successfully deleted
I20250112 02:52:34.923795 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 83c04f1a2cb04950a9f6170e252c19b4 (DEFAULT_TABLE table=default.Table_b1a770633f004b01b66ddfa83e0fd698 [id=20fddd79df734f3bb4552ac2ca4958e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.924340 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83c04f1a2cb04950a9f6170e252c19b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.927779 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 3499374810fb412e938325434c35618d (DEFAULT_TABLE table=default.Table_7870d93893b74b4b96d1a3809a9015e6 [id=4355589b5ccc4162ab8700e31173d1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.928232 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3499374810fb412e938325434c35618d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.932751 2036 tablet_replica.cc:331] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:34.932948 2036 raft_consensus.cc:2238] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:34.933094 2036 raft_consensus.cc:2267] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:34.933293 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d42f1611de49d6b6e5aaa9be907054" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:34.933588 2036 ts_tablet_manager.cc:1905] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:34.934011 855 leader_election.cc:343] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.934114 855 leader_election.cc:304] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:34.934655 855 leader_election.cc:343] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.934747 855 leader_election.cc:304] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:34.935055 2116 raft_consensus.cc:2743] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:34.935191 2116 raft_consensus.cc:2743] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:34.935626 855 leader_election.cc:343] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.935714 855 leader_election.cc:304] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:34.936156 2117 raft_consensus.cc:2743] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:34.936600 855 leader_election.cc:343] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:34.941084 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_5981a14692a2470290628d9e5934c4e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:34.941777 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5981a14692a2470290628d9e5934c4e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:34.950258 1016 tablet_service.cc:1467] Processing CreateTablet for tablet a8e7882e07784bdda18dc1ffd20963c8 (DEFAULT_TABLE table=default.Table_f3695b00a89b4353903587a6c1418d8c [id=d1781738b8a74f139b09863918cbb21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.950711 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8e7882e07784bdda18dc1ffd20963c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.953717 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 781a9943f705477cb6eae16df4468d2f (DEFAULT_TABLE table=default.Table_104ccd5f390f47a5878c67ea6de9d554 [id=872482df2ee94056a7d29ce227797f16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.954417 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 3d418cda9ed5449fa45690130cd6d5bc (DEFAULT_TABLE table=default.Table_8f2b912f93d84320b0f7e127c0543af6 [id=a49da46bf16343a488f96156139990a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.954797 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 781a9943f705477cb6eae16df4468d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.956697 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:34.959326 1026 tablet_service.cc:1467] Processing CreateTablet for tablet f3771166b77e4f90aaa116ab1a9dc299 (DEFAULT_TABLE table=default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.959535 1017 tablet_service.cc:1467] Processing CreateTablet for tablet a470f84d50df4cf0b4a1e5d248b4c32a (DEFAULT_TABLE table=default.Table_6e77252e817f43399c9b3f5b221795ad [id=1fa1474ab8804667b22e5f30c48c1630]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.959702 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3771166b77e4f90aaa116ab1a9dc299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.959681 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:34.960000 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a470f84d50df4cf0b4a1e5d248b4c32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.960340 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:34.960502 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet e3ee5d8a17874d7ba54de51568e7d2e6 (table default.Table_83177e7dc14645dbb0e5ef573e80e72f [id=2c077da077834ae4bcc84efdd9e89120]). Expected version 1 got 0
I20250112 02:52:34.960791 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet fccff64af8ca4400afc116db98918119 (table default.Table_15bcb2c0f2a74458a1bed231fd3437fc [id=ca9c8281bf254f248076cb600e7c4f87]). Expected version 2 got 0
I20250112 02:52:34.960987 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet f924459d1150472aafb325a11825db23 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]). Expected version 3 got 0
I20250112 02:52:34.961279 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
I20250112 02:52:34.961529 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet abfaad67d5e34c0a8338dccc0ef05466 (table default.Table_045e5b104dc64217aa39c22534007cab [id=2c0e94e8c5774da1b8130c7651d94500]). Expected version 1 got 0
I20250112 02:52:34.961741 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 09d06e2e8c13464db8cf801a292bdc99 (table default.Table_0bdc5f49d4ff4d6fb1290bc2dc0f70d9 [id=e3bb489c53454222a79702b798c0c2f1]). Expected version 2 got 0
I20250112 02:52:34.963779 2036 ts_tablet_manager.cc:1918] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:34.963877 2036 log.cc:1198] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/352c87f1dc724e7b94bc2e64abfce8d7
I20250112 02:52:34.964759 2036 ts_tablet_manager.cc:1939] T 352c87f1dc724e7b94bc2e64abfce8d7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:34.966081 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 352c87f1dc724e7b94bc2e64abfce8d7 (table default.Table_83212f303f0b4413aa6c62887da9dd34 [id=399c12e9075d4e55b21b17837ead62da]) successfully deleted
I20250112 02:52:34.966786 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 61eebc5f22664712af6e74e82809ca87 (DEFAULT_TABLE table=default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.967542 1016 tablet_service.cc:1467] Processing CreateTablet for tablet ea33053c62dc4707b759d5c1756ab4bd (DEFAULT_TABLE table=default.Table_489753b0c1154a4ba82ad30de84de181 [id=1b3a8812e2384219b500040a72efc1b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.968559 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d418cda9ed5449fa45690130cd6d5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.968911 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61eebc5f22664712af6e74e82809ca87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.970216 2036 tablet_replica.cc:331] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:34.970791 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:34.971081 2036 raft_consensus.cc:2238] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:34.971518 2036 raft_consensus.cc:2267] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:34.971647 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea33053c62dc4707b759d5c1756ab4bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:34.972376 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet d8eb0efc5d7f4c72bdfc7531a444d42a (table default.Table_d561e1740cc545dc8aa4babb76b45c37 [id=de27aeff6b3a49b3956498ea5015ff33]). Expected version 1 got 0
I20250112 02:52:34.972954 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 2c03b5e612d64314b210c11686fdfa2a (table default.Table_4934f6022f2441eda88751236a5798be [id=5c4769bfcefd4fe0a0f3c2e315d50c32]). Expected version 6 got 0
I20250112 02:52:34.973340 2036 ts_tablet_manager.cc:1905] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:34.975821 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 4916ae3917084cfe8c93240a8857be0e (table default.Table_69f363c4da234792886b544823b3214e [id=c7052afaf6284ef0972835c1f46cb829]). Expected version 1 got 0
I20250112 02:52:34.976640 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 23e7a2e88c374e949ced92c557225cbf (table default.Table_cd1f185d82b3474083246ec3702851fc [id=bfa9a504a8d34c86929efb0d15d049d0]). Expected version 3 got 0
I20250112 02:52:34.977175 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:34.978485 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:34.979177 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_b0af74e9ea8a4c5faea64696a16d5a9a" } new_table_name: "default.Table_96d1a09a902340ff8e05efb75faf3854" modify_external_catalogs: true
I20250112 02:52:34.981129 2117 raft_consensus.cc:491] T 763f77385e404c149c16edabe469c299 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.981246 2117 raft_consensus.cc:513] T 763f77385e404c149c16edabe469c299 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.982044 2167 raft_consensus.cc:491] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:34.982228 2117 leader_election.cc:290] T 763f77385e404c149c16edabe469c299 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:34.982317 2167 raft_consensus.cc:513] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:34.979195 448 catalog_manager.cc:5526] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:34.983173 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 5cc414d6420c41839621ab858067548b (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]). Expected version 1 got 0
I20250112 02:52:34.983485 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:34.983887 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:34.984163 2167 leader_election.cc:290] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:34.984498 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 7ea075826b16475ca474e284fc6bc545 (table default.Table_009a6976f6db4f218e3c3373368a58d7 [id=c5005669bb5f4c029d673bd39cc1535d]). Expected version 1 got 0
I20250112 02:52:34.985177 448 catalog_manager.cc:5526] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:34.990384 2036 ts_tablet_manager.cc:1918] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:34.990499 2036 log.cc:1198] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/1b184806b4884cd3b0a002905a1e57af
I20250112 02:52:34.990796 2036 ts_tablet_manager.cc:1939] T 1b184806b4884cd3b0a002905a1e57af P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:35.000895 1778 server_base.cc:1034] running on GCE node
I20250112 02:52:35.001111 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 42eed0a9f8354e5391ec2463dafdb683 (DEFAULT_TABLE table=default.Table_a5f330af613f4e5e9ecafdfe58eb0f1b [id=839e694f8f4f4424835ed91f8c604735]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:34.986290 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet f8c01b80fc8242d98fdfbf767212144d (table default.Table_63649e9607394f4bb72a3dd6e6d0dd69 [id=9807191b6e5c4d478cd47a54464fb9ce]). Expected version 1 got 0
I20250112 02:52:35.002048 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecf730b6985404a8134b497a4b792c8 (DEFAULT_TABLE table=default.Table_24fca5290fd54774bee413c9d84a997d [id=b9950eabab874da9a52a99555dfbefd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.002195 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 6117dc8b32a64535a2f4167a03bed0d2 (table default.Table_365739770a11434dbf260ea912e7ea81 [id=868adb627654450c936ef198e49d853b]). Expected version 1 got 0
I20250112 02:52:35.002358 1778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:52:35.002561 1019 tablet_service.cc:1467] Processing CreateTablet for tablet fc52271535df47e282bf1625c2e8e041 (DEFAULT_TABLE table=default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.002573 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 17c54c5dc86446cf85974777c8145ef6 (DEFAULT_TABLE table=default.Table_ab4d7c58924a4b8682b52637c23f125b [id=a622f7a973344a388b3275adbcce6234]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.003891 1031 tablet_service.cc:1467] Processing CreateTablet for tablet dcb7e0181cf74bbea6036c106516a7b3 (DEFAULT_TABLE table=default.Table_9d8f183a5347440f898ef33ee7299613 [id=c2e1172634dd4463b2a809de96e0450e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.004184 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 1b184806b4884cd3b0a002905a1e57af (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]) successfully deleted
I20250112 02:52:35.004595 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:35.007714 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet f31b777e7e0f43a28839d8c48c6a6590 (table default.Table_5fb592e41cc4406aa27546e7b41fdb88 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 1 got 0
I20250112 02:52:35.008339 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c69587c5a09d40f0baba5c2f6e8a7b41 (table default.Table_1216acc1fa6b4e51a55a89ffe22bcc9a [id=b5457e4343ee43618c8309e737b69ec3]). Expected version 1 got 0
I20250112 02:52:35.008843 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
W20250112 02:52:35.009552 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 7a34d03a80ae402db6607261a5c40b60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a34d03a80ae402db6607261a5c40b60 already in progress: creating tablet
I20250112 02:52:35.010129 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:35.010730 2036 tablet_replica.cc:331] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:35.011200 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:35.011624 2036 raft_consensus.cc:2238] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:35.012132 2036 raft_consensus.cc:2267] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:35.013324 2036 ts_tablet_manager.cc:1905] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:35.013928 448 catalog_manager.cc:5526] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:35.017375 1181 tablet_bootstrap.cc:492] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:35.017524 1181 ts_tablet_manager.cc:1397] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.314s user 0.011s sys 0.033s
I20250112 02:52:35.018815 1181 raft_consensus.cc:357] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.019109 1181 raft_consensus.cc:383] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:35.019222 1181 raft_consensus.cc:738] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:35.019645 1181 consensus_queue.cc:260] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.022030 1778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:52:35.023299 1778 hybrid_clock.cc:648] HybridClock initialized: now 1736650355023190 us; error 64 us; skew 500 ppm
I20250112 02:52:35.023730 1778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:52:35.023682 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8daf4ef3593410eae87672402292be6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:35.024127 746 tablet_service.cc:1467] Processing CreateTablet for tablet 3499374810fb412e938325434c35618d (DEFAULT_TABLE table=default.Table_7870d93893b74b4b96d1a3809a9015e6 [id=4355589b5ccc4162ab8700e31173d1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.024300 1181 ts_tablet_manager.cc:1428] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250112 02:52:35.024528 1181 tablet_bootstrap.cc:492] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:35.025547 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3499374810fb412e938325434c35618d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.026006 1181 tablet_bootstrap.cc:654] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:35.027077 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 7cd9985cd20148f195a857c478b0755f (table default.Table_9b65dd291c9f4fcdaa6d5e37f04d24ae [id=e63daf724d874c89badafb8d59653f74]). Expected version 1 got 0
I20250112 02:52:34.939337 756 tablet_service.cc:1467] Processing CreateTablet for tablet 83c04f1a2cb04950a9f6170e252c19b4 (DEFAULT_TABLE table=default.Table_b1a770633f004b01b66ddfa83e0fd698 [id=20fddd79df734f3bb4552ac2ca4958e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.031302 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:35.031618 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
I20250112 02:52:35.032439 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:35.032608 448 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:35.032692 750 tablet_service.cc:1467] Processing CreateTablet for tablet 54d3664071b146a3ac7c6885f9bda789 (DEFAULT_TABLE table=default.Table_787c491d6a594a27b46814669b3ccc2b [id=e14369e145934ad9b3c515b33ad29a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.032994 755 tablet_service.cc:1467] Processing CreateTablet for tablet d95f98ae7bdf42518bbff55e257fea88 (DEFAULT_TABLE table=default.Table_dd77b8845fdd46b7a2debbadaf553f75 [id=70c61753e9944e1fbd65e9ec41a48ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.033890 749 tablet_service.cc:1467] Processing CreateTablet for tablet 3efcd32b4ccc4fada5adb3ca3bcdddaa (DEFAULT_TABLE table=default.Table_28b2d14acaa04d2a90164800ca8ced3c [id=afa0a72f5c634b289342dc0872945868]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:35.037119 855 leader_election.cc:343] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:35.037216 855 leader_election.cc:304] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:35.037539 2117 raft_consensus.cc:2743] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:35.039443 2036 ts_tablet_manager.cc:1918] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 02:52:35.043293 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5804fc4130f84c69832c1022b7bd7fa5 consensus state has no leader
I20250112 02:52:35.045987 759 tablet_service.cc:1467] Processing CreateTablet for tablet 2d47f6d663934b25a0200ee9fa81165d (DEFAULT_TABLE table=default.Table_fbc52174b22b4aafa4ede40120b90daf [id=4418eb326c7342e6a8c16a263827adbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.047890 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83c04f1a2cb04950a9f6170e252c19b4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:35.050452 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 377668us
I20250112 02:52:35.050524 744 tablet_service.cc:1467] Processing CreateTablet for tablet 3d418cda9ed5449fa45690130cd6d5bc (DEFAULT_TABLE table=default.Table_8f2b912f93d84320b0f7e127c0543af6 [id=a49da46bf16343a488f96156139990a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.053515 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3efcd32b4ccc4fada5adb3ca3bcdddaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.053398 2159 tablet.cc:1721] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:35.054163 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763f77385e404c149c16edabe469c299" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:35.052551 2158 tablet.cc:1721] T 7ea075826b16475ca474e284fc6bc545 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:35.052572 2160 tablet.cc:1721] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:35.055333 2036 log.cc:1198] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/9d5b3df88a0c45dba879b3d506710614
I20250112 02:52:35.056771 2036 ts_tablet_manager.cc:1939] T 9d5b3df88a0c45dba879b3d506710614 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
W20250112 02:52:35.054734 855 leader_election.cc:343] T 763f77385e404c149c16edabe469c299 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:35.058902 1181 tablet_bootstrap.cc:492] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:35.059052 1181 ts_tablet_manager.cc:1397] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20250112 02:52:35.060652 2056 raft_consensus.cc:491] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:35.061410 2167 raft_consensus.cc:491] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:35.062148 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 9d5b3df88a0c45dba879b3d506710614 (table default.Table_49b5979e0b174043b61f2f1d61a5b58c [id=b81cf271b11e45dcb0b6bce102cd7a0c]) successfully deleted
I20250112 02:52:35.064201 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42eed0a9f8354e5391ec2463dafdb683. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.064960 1181 raft_consensus.cc:357] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.065220 1181 raft_consensus.cc:383] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:35.065924 2167 raft_consensus.cc:513] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.067778 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ecf730b6985404a8134b497a4b792c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.067824 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 5450777104da48f1821575658fd31afd (DEFAULT_TABLE table=default.Table_af9d530db1774d9eafed6727be07aeaf [id=14b00b3f93ef44269cf7de795ab41492]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.068230 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5450777104da48f1821575658fd31afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.068934 1181 raft_consensus.cc:738] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:35.069182 1181 consensus_queue.cc:260] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.069703 1181 ts_tablet_manager.cc:1428] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.005s user 0.000s sys 0.000s
I20250112 02:52:35.063192 2036 tablet_replica.cc:331] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:35.070523 1181 tablet_bootstrap.cc:492] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:35.070911 2056 raft_consensus.cc:513] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.071750 2056 leader_election.cc:290] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:35.073170 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d95f98ae7bdf42518bbff55e257fea88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.076655 1181 tablet_bootstrap.cc:654] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:35.077028 2172 tablet.cc:1721] T 7ea075826b16475ca474e284fc6bc545 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:35.080870 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d47f6d663934b25a0200ee9fa81165d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.083369 2036 raft_consensus.cc:2238] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:35.083539 2036 raft_consensus.cc:2267] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:35.083987 2036 ts_tablet_manager.cc:1905] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:35.085157 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d418cda9ed5449fa45690130cd6d5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.095400 1181 tablet_bootstrap.cc:492] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:35.095526 1181 ts_tablet_manager.cc:1397] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20250112 02:52:35.096541 1181 raft_consensus.cc:357] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.096694 1181 raft_consensus.cc:383] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:35.096750 1181 raft_consensus.cc:738] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:35.096928 1181 consensus_queue.cc:260] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.098368 1181 ts_tablet_manager.cc:1428] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 02:52:35.098613 1181 tablet_bootstrap.cc:492] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:35.108067 1181 tablet_bootstrap.cc:654] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:35.112289 854 leader_election.cc:343] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:35.112382 854 leader_election.cc:304] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:35.112875 2117 raft_consensus.cc:2743] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:35.115087 2024 ts_tablet_manager.cc:1918] T 161fe26b9c0e4543b7aeb0021506d72a P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:35.115545 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 76009bac22ce4136b148b493a004a0f7 (DEFAULT_TABLE table=default.Table_ea881fde8d694f71b5faf60948630d96 [id=ce171e0d83bd436eaad276bedf32030e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.122121 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 93aba9b4d1dd485cb62c4827c916a1d5 (DEFAULT_TABLE table=default.Table_d5f4d4c431d24cc0ba82a7dcd012368b [id=3059ab365aff4430b1570224188486fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.124322 2167 leader_election.cc:290] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:35.124457 889 tablet_service.cc:1467] Processing CreateTablet for tablet 29d7774d4a284d24b833d50809996fc2 (DEFAULT_TABLE table=default.Table_84016330aad94c2ba3aec196a69d28de [id=84d89da1bbaf4f5985335bb97a9f6d86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.124776 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29d7774d4a284d24b833d50809996fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.125756 2024 log.cc:1198] T 161fe26b9c0e4543b7aeb0021506d72a P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/161fe26b9c0e4543b7aeb0021506d72a
I20250112 02:52:35.126360 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17c54c5dc86446cf85974777c8145ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.129317 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76009bac22ce4136b148b493a004a0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.130479 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93aba9b4d1dd485cb62c4827c916a1d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.131807 889 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe57d374eb44c3fb1532484e214b654 (DEFAULT_TABLE table=default.Table_6730101812ca4bca8e85617d952cc549 [id=45403aee8fed439c94e3b7d3c8898668]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.132180 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fe57d374eb44c3fb1532484e214b654. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.134022 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcb7e0181cf74bbea6036c106516a7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.134855 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc52271535df47e282bf1625c2e8e041. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.138383 899 tablet_service.cc:1467] Processing CreateTablet for tablet 0fea8f6f13d54945934ee3880e00c5e9 (DEFAULT_TABLE table=default.Table_f38c3f9c44b741219f03cf46cfbc9937 [id=a41ea1ae71784a99a0ca89a45825cb85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.138656 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fea8f6f13d54945934ee3880e00c5e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.144308 2036 ts_tablet_manager.cc:1918] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:35.144851 2036 log.cc:1198] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/a85300a9aa1c4c50a3957a004bea7933
I20250112 02:52:35.149403 889 tablet_service.cc:1467] Processing CreateTablet for tablet f52b838214c54ac69ad30dac6e56669a (DEFAULT_TABLE table=default.Table_2989b0fb83514c45ac9836e95e444f31 [id=f612d417385a4a02814d993aab77b5a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.149706 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f52b838214c54ac69ad30dac6e56669a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.150869 2036 ts_tablet_manager.cc:1939] T a85300a9aa1c4c50a3957a004bea7933 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:35.152010 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet a85300a9aa1c4c50a3957a004bea7933 (table default.Table_679c07c6f25644c6a4136aa56dd5100a [id=3a4127ef6ed145d4bdbb562adadfbb5c]) successfully deleted
I20250112 02:52:35.153054 2024 ts_tablet_manager.cc:1939] T 161fe26b9c0e4543b7aeb0021506d72a P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:35.153564 2024 tablet_replica.cc:331] T b9199b0d2b924a79938a979df4196bb4 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:35.154173 899 tablet_service.cc:1467] Processing CreateTablet for tablet 5b741e4f6abd4695b3ba1d7a667b6046 (DEFAULT_TABLE table=default.Table_e781b72b17d54bf8ba293cd013e5e14d [id=332595f0d3a24adf8b9bbe47b82b9f66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.154223 2024 raft_consensus.cc:2238] T b9199b0d2b924a79938a979df4196bb4 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:35.154438 2024 raft_consensus.cc:2267] T b9199b0d2b924a79938a979df4196bb4 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:35.154547 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b741e4f6abd4695b3ba1d7a667b6046. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.160290 2036 tablet_replica.cc:331] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:35.160447 2036 raft_consensus.cc:2238] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:35.160557 2036 raft_consensus.cc:2267] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:35.160962 2036 ts_tablet_manager.cc:1905] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:35.161120 2024 ts_tablet_manager.cc:1905] T b9199b0d2b924a79938a979df4196bb4 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:35.168900 1181 tablet_bootstrap.cc:492] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:35.169019 1181 ts_tablet_manager.cc:1397] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.000s
I20250112 02:52:35.169625 1181 raft_consensus.cc:357] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.169780 1181 raft_consensus.cc:383] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:35.169821 1181 raft_consensus.cc:738] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:35.170011 1181 consensus_queue.cc:260] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.170549 1181 ts_tablet_manager.cc:1428] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250112 02:52:35.170753 1181 tablet_bootstrap.cc:492] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:35.173097 899 tablet_service.cc:1467] Processing CreateTablet for tablet 7c1ae3f34b434ee5bc3519cce8d283a6 (DEFAULT_TABLE table=default.Table_b236e732be7844a08eaa268cb451d22f [id=48b14376b80643f5a6d8e58df085d8f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.173411 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c1ae3f34b434ee5bc3519cce8d283a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.177321 1181 tablet_bootstrap.cc:654] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:35.179699 890 tablet_service.cc:1467] Processing CreateTablet for tablet 56ddc025a8314f6f84d229191eb56692 (DEFAULT_TABLE table=default.Table_b2879d6a85294514989d7b68d3ac97d4 [id=2ddd5dd296b94bc8b96a97b66e775836]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.179996 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56ddc025a8314f6f84d229191eb56692. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.186651 890 tablet_service.cc:1467] Processing CreateTablet for tablet 99407ead9cf547e3bbe78c0d0b327e6a (DEFAULT_TABLE table=default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.187048 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99407ead9cf547e3bbe78c0d0b327e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.192775 892 tablet_service.cc:1467] Processing CreateTablet for tablet cbe30f9860e2492b90e8cedf8ff9daba (DEFAULT_TABLE table=default.Table_75f163a2f4aa48df95c9bafd3b311f4b [id=e6546a8535524b5996680a4e5451032f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.193054 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbe30f9860e2492b90e8cedf8ff9daba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.193835 898 tablet_service.cc:1467] Processing CreateTablet for tablet a1123077650c4e89bffddd6d1264e428 (DEFAULT_TABLE table=default.Table_36b06d2edb8e4d23a3cb7021a8267db4 [id=3506f201d2674e38a353c87e1b2f26a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.194099 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1123077650c4e89bffddd6d1264e428. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.203837 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:35.204227 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
I20250112 02:52:35.204574 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5cc414d6420c41839621ab858067548b (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]). Expected version 1 got 0
I20250112 02:52:35.205451 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:35.207634 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet e3ee5d8a17874d7ba54de51568e7d2e6 (table default.Table_83177e7dc14645dbb0e5ef573e80e72f [id=2c077da077834ae4bcc84efdd9e89120]). Expected version 1 got 0
I20250112 02:52:35.208657 890 tablet_service.cc:1467] Processing CreateTablet for tablet 5524d75599f04becbe6a94426adb7026 (DEFAULT_TABLE table=default.Table_ef2001a2d9b24b4bb1f8fb9102703022 [id=b7739f28885a453d9ef3966511183c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.208971 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5524d75599f04becbe6a94426adb7026. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.210048 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:35.210516 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet fccff64af8ca4400afc116db98918119 (table default.Table_15bcb2c0f2a74458a1bed231fd3437fc [id=ca9c8281bf254f248076cb600e7c4f87]). Expected version 2 got 0
I20250112 02:52:35.210731 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet f924459d1150472aafb325a11825db23 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]). Expected version 3 got 0
I20250112 02:52:35.211627 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:35.212409 899 tablet_service.cc:1467] Processing CreateTablet for tablet f559dedfb7854cf890bef362c50b4213 (DEFAULT_TABLE table=default.Table_7921b99e24f84cceb58128f18925ad49 [id=0525f87eccbb4a1d8726d18a00f9a4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.212718 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f559dedfb7854cf890bef362c50b4213. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.217867 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet abfaad67d5e34c0a8338dccc0ef05466 (table default.Table_045e5b104dc64217aa39c22534007cab [id=2c0e94e8c5774da1b8130c7651d94500]). Expected version 1 got 0
I20250112 02:52:35.218230 1181 tablet_bootstrap.cc:492] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:35.218763 1181 ts_tablet_manager.cc:1397] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20250112 02:52:35.219728 2024 ts_tablet_manager.cc:1918] T b9199b0d2b924a79938a979df4196bb4 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:35.219869 889 tablet_service.cc:1467] Processing CreateTablet for tablet 2796289c7e8f4b45b26508c50b57c640 (DEFAULT_TABLE table=default.Table_23c987a24fed4ceeb90fd25f0367b08f [id=a9f367de65284002a6a8ea94fc8cb7d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.220156 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2796289c7e8f4b45b26508c50b57c640. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.220171 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 09d06e2e8c13464db8cf801a292bdc99 (table default.Table_0bdc5f49d4ff4d6fb1290bc2dc0f70d9 [id=e3bb489c53454222a79702b798c0c2f1]). Expected version 2 got 0
I20250112 02:52:35.224296 754 tablet_service.cc:1467] Processing CreateTablet for tablet 781a9943f705477cb6eae16df4468d2f (DEFAULT_TABLE table=default.Table_104ccd5f390f47a5878c67ea6de9d554 [id=872482df2ee94056a7d29ce227797f16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.226384 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6117dc8b32a64535a2f4167a03bed0d2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:35.226565 916 raft_consensus.cc:2463] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:35.226583 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet d8eb0efc5d7f4c72bdfc7531a444d42a (table default.Table_d561e1740cc545dc8aa4babb76b45c37 [id=de27aeff6b3a49b3956498ea5015ff33]). Expected version 1 got 0
I20250112 02:52:35.227744 898 tablet_service.cc:1467] Processing CreateTablet for tablet 37d38f5e525440f8a0f7822620933435 (DEFAULT_TABLE table=default.Table_39f67a35c5c640dd8cbbcea89df5e4fc [id=9d267774bf59428e854651956fbd79be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.228492 1181 raft_consensus.cc:357] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.228631 1181 raft_consensus.cc:383] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:35.228672 1181 raft_consensus.cc:738] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:35.230271 2024 log.cc:1198] T b9199b0d2b924a79938a979df4196bb4 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/b9199b0d2b924a79938a979df4196bb4
I20250112 02:52:35.230623 2167 raft_consensus.cc:491] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:35.230674 2094 tablet.cc:1721] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:35.230999 1177 tablet_bootstrap.cc:492] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:35.231791 2003 ts_tablet_manager.cc:1918] T aadb2ab1d2024f889841ac91e065e064 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:35.232069 1181 consensus_queue.cc:260] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.232622 1181 ts_tablet_manager.cc:1428] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20250112 02:52:35.233067 1181 tablet_bootstrap.cc:492] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:35.233726 2167 raft_consensus.cc:513] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.234477 2167 leader_election.cc:290] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:35.234629 1181 tablet_bootstrap.cc:654] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:35.239074 1177 ts_tablet_manager.cc:1397] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.626s user 0.011s sys 0.007s
I20250112 02:52:35.240080 1177 raft_consensus.cc:357] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.240311 1177 raft_consensus.cc:383] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:35.240515 1177 raft_consensus.cc:738] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:35.240741 1177 consensus_queue.cc:260] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.241261 1177 ts_tablet_manager.cc:1428] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:35.241336 2003 log.cc:1198] T aadb2ab1d2024f889841ac91e065e064 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/aadb2ab1d2024f889841ac91e065e064
I20250112 02:52:35.241474 2117 raft_consensus.cc:491] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:35.241709 2003 ts_tablet_manager.cc:1939] T aadb2ab1d2024f889841ac91e065e064 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:35.241595 2117 raft_consensus.cc:513] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.242213 2117 leader_election.cc:290] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:35.242363 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 27) took 1165 ms. Trace:
I20250112 02:52:35.242447 2003 rpcz_store.cc:276] 0112 02:52:34.076684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.076724 (+ 40us) service_pool.cc:225] Handling call
0112 02:52:34.448374 (+371650us) ts_tablet_manager.cc:1145] Deleting tablet aadb2ab1d2024f889841ac91e065e064
0112 02:52:34.448381 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:34.600751 (+152370us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.081968 (+481217us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.242309 (+160341us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37373,"tablet-delete.queue_time_us":362200}
I20250112 02:52:35.242754 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a34d03a80ae402db6607261a5c40b60" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:35.242902 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 25) took 1171 ms. Trace:
I20250112 02:52:35.242970 2003 rpcz_store.cc:276] 0112 02:52:34.071763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.071860 (+ 97us) service_pool.cc:225] Handling call
0112 02:52:35.242774 (+1170914us) ts_tablet_manager.cc:1145] Deleting tablet 416fb6820cd54e118950c68d381f51b2
0112 02:52:35.242782 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:35.242854 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1155897}
I20250112 02:52:35.243253 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 29) took 1156 ms. Trace:
W20250112 02:52:35.243243 855 leader_election.cc:343] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:35.243327 2003 rpcz_store.cc:276] 0112 02:52:34.086312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.086415 (+ 103us) service_pool.cc:225] Handling call
0112 02:52:35.243125 (+1156710us) ts_tablet_manager.cc:1145] Deleting tablet 70715db0394d46d9b6b289160b00be7d
0112 02:52:35.243131 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:35.243209 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1155994}
I20250112 02:52:35.243554 2003 tablet_replica.cc:331] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
W20250112 02:52:35.244118 1049 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.009s sys 0.005s
I20250112 02:52:35.244796 2024 ts_tablet_manager.cc:1939] T b9199b0d2b924a79938a979df4196bb4 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:35.246101 741 tablet_service.cc:1467] Processing CreateTablet for tablet fc52271535df47e282bf1625c2e8e041 (DEFAULT_TABLE table=default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.247692 745 tablet_service.cc:1467] Processing CreateTablet for tablet f3771166b77e4f90aaa116ab1a9dc299 (DEFAULT_TABLE table=default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.248109 758 tablet_service.cc:1467] Processing CreateTablet for tablet a470f84d50df4cf0b4a1e5d248b4c32a (DEFAULT_TABLE table=default.Table_6e77252e817f43399c9b3f5b221795ad [id=1fa1474ab8804667b22e5f30c48c1630]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.248193 1173 tablet_bootstrap.cc:492] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:35.248342 1173 ts_tablet_manager.cc:1397] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.579s user 0.005s sys 0.009s
I20250112 02:52:35.248698 760 tablet_service.cc:1467] Processing CreateTablet for tablet 5450777104da48f1821575658fd31afd (DEFAULT_TABLE table=default.Table_af9d530db1774d9eafed6727be07aeaf [id=14b00b3f93ef44269cf7de795ab41492]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.249076 757 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecf730b6985404a8134b497a4b792c8 (DEFAULT_TABLE table=default.Table_24fca5290fd54774bee413c9d84a997d [id=b9950eabab874da9a52a99555dfbefd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.250849 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet aadb2ab1d2024f889841ac91e065e064 (table default.Table_6255caf5a65d4d9192e68f39413398f0 [id=5e4665095f4947bd8adb7ec66be4b456]) successfully deleted
W20250112 02:52:35.251072 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 416fb6820cd54e118950c68d381f51b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 416fb6820cd54e118950c68d381f51b2 already in progress: creating tablet
W20250112 02:52:35.251287 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 70715db0394d46d9b6b289160b00be7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70715db0394d46d9b6b289160b00be7d already in progress: creating tablet
W20250112 02:52:35.262252 1049 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.553s user 0.010s sys 0.005s
I20250112 02:52:35.262871 2167 consensus_queue.cc:1035] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:52:35.263530 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54d3664071b146a3ac7c6885f9bda789. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.264183 1177 tablet_bootstrap.cc:492] T 6117dc8b32a64535a2f4167a03bed0d2 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:35.257426 751 tablet_service.cc:1467] Processing CreateTablet for tablet 61eebc5f22664712af6e74e82809ca87 (DEFAULT_TABLE table=default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.276229 2159 tablet.cc:1721] T 7cd9985cd20148f195a857c478b0755f P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:35.276906 1778 webserver.cc:458] Webserver started at http://127.0.104.61:33823/ using document root <none> and password file <none>
I20250112 02:52:35.278854 747 tablet_service.cc:1467] Processing CreateTablet for tablet ea33053c62dc4707b759d5c1756ab4bd (DEFAULT_TABLE table=default.Table_489753b0c1154a4ba82ad30de84de181 [id=1b3a8812e2384219b500040a72efc1b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.279601 743 tablet_service.cc:1467] Processing CreateTablet for tablet a8e7882e07784bdda18dc1ffd20963c8 (DEFAULT_TABLE table=default.Table_f3695b00a89b4353903587a6c1418d8c [id=d1781738b8a74f139b09863918cbb21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.282362 1778 fs_manager.cc:362] Metadata directory not provided
I20250112 02:52:35.282895 1778 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20250112 02:52:35.286245 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 233407us
I20250112 02:52:35.286863 753 tablet_service.cc:1467] Processing CreateTablet for tablet 42eed0a9f8354e5391ec2463dafdb683 (DEFAULT_TABLE table=default.Table_a5f330af613f4e5e9ecafdfe58eb0f1b [id=839e694f8f4f4424835ed91f8c604735]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.287179 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc52271535df47e282bf1625c2e8e041. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.288870 2158 tablet.cc:1721] T f8c01b80fc8242d98fdfbf767212144d P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:35.289620 2160 tablet.cc:1721] T d8eb0efc5d7f4c72bdfc7531a444d42a P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:35.289921 2190 raft_consensus.cc:491] T 89b70dce4d1d4e29997a18297ac0f82a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:35.293823 752 tablet_service.cc:1467] Processing CreateTablet for tablet 17c54c5dc86446cf85974777c8145ef6 (DEFAULT_TABLE table=default.Table_ab4d7c58924a4b8682b52637c23f125b [id=a622f7a973344a388b3275adbcce6234]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:35.297315 713 consensus_peers.cc:487] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a85300a9aa1c4c50a3957a004bea7933. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:35.297407 2190 raft_consensus.cc:513] T 89b70dce4d1d4e29997a18297ac0f82a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.297732 989 leader_election.cc:304] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:35.303536 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b70dce4d1d4e29997a18297ac0f82a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:35.303597 2191 tablet.cc:1721] T d8eb0efc5d7f4c72bdfc7531a444d42a P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:35.303768 918 raft_consensus.cc:2388] T 89b70dce4d1d4e29997a18297ac0f82a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 289afc116aa74408aa5bff74e11c50d8 in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
I20250112 02:52:35.304847 1778 fs_manager.cc:714] Time spent opening directory manager: real 0.013s user 0.002s sys 0.008s
I20250112 02:52:35.305071 2190 leader_election.cc:290] T 89b70dce4d1d4e29997a18297ac0f82a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:35.309561 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a470f84d50df4cf0b4a1e5d248b4c32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.314474 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ecf730b6985404a8134b497a4b792c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.314503 1177 tablet_bootstrap.cc:654] T 6117dc8b32a64535a2f4167a03bed0d2 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:35.318639 989 leader_election.cc:343] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 1dda52f012794493a102ed1abc718f5b
I20250112 02:52:35.319226 1727 raft_consensus.cc:2798] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:35.321879 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5450777104da48f1821575658fd31afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.325307 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 781a9943f705477cb6eae16df4468d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.326263 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3771166b77e4f90aaa116ab1a9dc299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.341202 2205 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:52:35.343380 1778 fs_manager.cc:730] Time spent opening block manager: real 0.036s user 0.010s sys 0.023s
I20250112 02:52:35.343595 1778 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/data,/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/wal
uuid: "f4a49a5756b143fcb767c3f41463f8d9"
format_stamp: "Formatted at 2025-01-12 02:52:20 on dist-test-slave-h743"
I20250112 02:52:35.344820 1778 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:52:35.347100 2024 tablet_replica.cc:331] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:35.355127 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61eebc5f22664712af6e74e82809ca87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.366958 748 tablet_service.cc:1467] Processing CreateTablet for tablet dcb7e0181cf74bbea6036c106516a7b3 (DEFAULT_TABLE table=default.Table_9d8f183a5347440f898ef33ee7299613 [id=c2e1172634dd4463b2a809de96e0450e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.368633 2003 raft_consensus.cc:2238] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:35.371398 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17c54c5dc86446cf85974777c8145ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.371439 756 tablet_service.cc:1467] Processing CreateTablet for tablet 76009bac22ce4136b148b493a004a0f7 (DEFAULT_TABLE table=default.Table_ea881fde8d694f71b5faf60948630d96 [id=ce171e0d83bd436eaad276bedf32030e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.374240 746 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe57d374eb44c3fb1532484e214b654 (DEFAULT_TABLE table=default.Table_6730101812ca4bca8e85617d952cc549 [id=45403aee8fed439c94e3b7d3c8898668]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.375360 759 tablet_service.cc:1467] Processing CreateTablet for tablet 93aba9b4d1dd485cb62c4827c916a1d5 (DEFAULT_TABLE table=default.Table_d5f4d4c431d24cc0ba82a7dcd012368b [id=3059ab365aff4430b1570224188486fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.375996 2117 raft_consensus.cc:491] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:35.376178 2117 raft_consensus.cc:513] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.376986 2117 leader_election.cc:290] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:35.379185 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5b3df88a0c45dba879b3d506710614" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:35.379349 772 raft_consensus.cc:3054] T 9d5b3df88a0c45dba879b3d506710614 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:35.383757 2003 raft_consensus.cc:2267] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:35.372009 755 tablet_service.cc:1467] Processing CreateTablet for tablet 29d7774d4a284d24b833d50809996fc2 (DEFAULT_TABLE table=default.Table_84016330aad94c2ba3aec196a69d28de [id=84d89da1bbaf4f5985335bb97a9f6d86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.389842 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dda52f012794493a102ed1abc718f5b" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:35.392069 2003 ts_tablet_manager.cc:1905] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:35.390939 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7320ac55dbfa47fe851fff94d32e1ffe" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:35.395486 855 leader_election.cc:343] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:35.399117 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76009bac22ce4136b148b493a004a0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.319664 2207 consensus_queue.cc:1035] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20250112 02:52:35.348498 2050 consensus_queue.cc:1035] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.088s
I20250112 02:52:35.388633 2167 consensus_queue.cc:1035] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
W20250112 02:52:35.388830 989 leader_election.cc:343] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 1b184806b4884cd3b0a002905a1e57af
I20250112 02:52:35.406538 1177 tablet_bootstrap.cc:492] T 6117dc8b32a64535a2f4167a03bed0d2 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:35.406744 1177 ts_tablet_manager.cc:1397] T 6117dc8b32a64535a2f4167a03bed0d2 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.163s user 0.015s sys 0.008s
I20250112 02:52:35.407910 1177 raft_consensus.cc:357] T 6117dc8b32a64535a2f4167a03bed0d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.408186 1177 raft_consensus.cc:383] T 6117dc8b32a64535a2f4167a03bed0d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:35.408309 1177 raft_consensus.cc:738] T 6117dc8b32a64535a2f4167a03bed0d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:35.408583 1177 consensus_queue.cc:260] T 6117dc8b32a64535a2f4167a03bed0d2 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.409386 1177 ts_tablet_manager.cc:1428] T 6117dc8b32a64535a2f4167a03bed0d2 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250112 02:52:35.410581 1177 tablet_bootstrap.cc:492] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:35.412907 741 tablet_service.cc:1467] Processing CreateTablet for tablet 0fea8f6f13d54945934ee3880e00c5e9 (DEFAULT_TABLE table=default.Table_f38c3f9c44b741219f03cf46cfbc9937 [id=a41ea1ae71784a99a0ca89a45825cb85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.417629 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8e7882e07784bdda18dc1ffd20963c8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:35.428035 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 48307.9us
I20250112 02:52:35.444099 2185 tablet.cc:1721] T e3ee5d8a17874d7ba54de51568e7d2e6 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:35.446381 745 tablet_service.cc:1467] Processing CreateTablet for tablet 5b741e4f6abd4695b3ba1d7a667b6046 (DEFAULT_TABLE table=default.Table_e781b72b17d54bf8ba293cd013e5e14d [id=332595f0d3a24adf8b9bbe47b82b9f66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.450148 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352c87f1dc724e7b94bc2e64abfce8d7" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:35.450851 749 tablet_service.cc:1467] Processing CreateTablet for tablet 7c1ae3f34b434ee5bc3519cce8d283a6 (DEFAULT_TABLE table=default.Table_b236e732be7844a08eaa268cb451d22f [id=48b14376b80643f5a6d8e58df085d8f1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:35.458163 989 leader_election.cc:343] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 352c87f1dc724e7b94bc2e64abfce8d7
I20250112 02:52:35.465166 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93aba9b4d1dd485cb62c4827c916a1d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.468015 2172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:49624 (request call id 32) took 1373 ms. Trace:
I20250112 02:52:35.470443 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b184806b4884cd3b0a002905a1e57af" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:35.473555 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6117dc8b32a64535a2f4167a03bed0d2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:35.473802 774 raft_consensus.cc:2463] T 6117dc8b32a64535a2f4167a03bed0d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:35.473817 742 tablet_service.cc:1467] Processing CreateTablet for tablet 56ddc025a8314f6f84d229191eb56692 (DEFAULT_TABLE table=default.Table_b2879d6a85294514989d7b68d3ac97d4 [id=2ddd5dd296b94bc8b96a97b66e775836]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.474478 744 tablet_service.cc:1467] Processing CreateTablet for tablet f52b838214c54ac69ad30dac6e56669a (DEFAULT_TABLE table=default.Table_2989b0fb83514c45ac9836e95e444f31 [id=f612d417385a4a02814d993aab77b5a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.475276 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b741e4f6abd4695b3ba1d7a667b6046. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.476328 2191 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:49624 (request call id 46) took 1344 ms. Trace:
I20250112 02:52:35.476406 754 tablet_service.cc:1467] Processing CreateTablet for tablet 99407ead9cf547e3bbe78c0d0b327e6a (DEFAULT_TABLE table=default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.478454 2094 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:49624 (request call id 39) took 1358 ms. Trace:
I20250112 02:52:35.478719 989 leader_election.cc:304] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:35.478968 2207 raft_consensus.cc:2798] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250112 02:52:35.479027 989 leader_election.cc:343] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 1726d61cc36a4deb83c5d604e8daf4f0
I20250112 02:52:35.479010 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ee98f055024429eafaa29bb961cab87" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:35.488973 965 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:35.478562 2094 rpcz_store.cc:276] 0112 02:52:34.114314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.114414 (+ 100us) service_pool.cc:225] Handling call
0112 02:52:35.472859 (+1358445us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:34.276615 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 02:52:34.305031 (+ 28416us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 02:52:34.305041 (+ 10us) alter_schema_op.cc:66] Acquired schema lock
0112 02:52:34.305047 (+ 6us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 02:52:34.305080 (+ 33us) alter_schema_op.cc:128] START. Timestamp: 1736650354305031
0112 02:52:34.305085 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:34.308238 (+ 3153us) log.cc:844] Serialized 173 byte log entry
0112 02:52:34.888562 (+580324us) op_driver.cc:464] REPLICATION: finished
0112 02:52:35.230641 (+342079us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 02:52:35.333230 (+102589us) log.cc:844] Serialized 5 byte log entry
0112 02:52:35.469533 (+136303us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.472791 (+ 3258us) log.cc:844] Serialized 14 byte log entry
0112 02:52:35.472829 (+ 38us) alter_schema_op.cc:73] Released schema lock
0112 02:52:35.472832 (+ 3us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":342046,"mutex_wait_us":22704,"prepare.queue_time_us":159279,"prepare.run_cpu_time_us":13655,"prepare.run_wall_time_us":346819,"raft.queue_time_us":22785,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":258,"replication_time_us":583454,"thread_start_us":1078516,"threads_started":3,"wal-append.queue_time_us":299425}]]}
I20250112 02:52:35.504938 772 raft_consensus.cc:2463] T 9d5b3df88a0c45dba879b3d506710614 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:35.507651 2190 raft_consensus.cc:491] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:35.507714 758 tablet_service.cc:1467] Processing CreateTablet for tablet cbe30f9860e2492b90e8cedf8ff9daba (DEFAULT_TABLE table=default.Table_75f163a2f4aa48df95c9bafd3b311f4b [id=e6546a8535524b5996680a4e5451032f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.508638 2190 raft_consensus.cc:513] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.509377 2190 leader_election.cc:290] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:35.511674 2094 tablet.cc:1721] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:52:35.512126 713 leader_election.cc:343] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 73136be25c79491784d3ea71aa9b27f2
I20250112 02:52:35.513789 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29d7774d4a284d24b833d50809996fc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.514058 2159 tablet.cc:1721] T 4916ae3917084cfe8c93240a8857be0e P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:35.514647 750 tablet_service.cc:1467] Processing CreateTablet for tablet 5524d75599f04becbe6a94426adb7026 (DEFAULT_TABLE table=default.Table_ef2001a2d9b24b4bb1f8fb9102703022 [id=b7739f28885a453d9ef3966511183c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.504894 2191 rpcz_store.cc:276] 0112 02:52:34.130694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.131370 (+ 676us) service_pool.cc:225] Handling call
0112 02:52:35.475000 (+1343630us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:34.350994 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 02:52:34.351086 (+ 92us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 02:52:34.351095 (+ 9us) alter_schema_op.cc:66] Acquired schema lock
0112 02:52:34.351101 (+ 6us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 02:52:34.351146 (+ 45us) alter_schema_op.cc:128] START. Timestamp: 1736650354351076
0112 02:52:34.351149 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:34.351294 (+ 145us) log.cc:844] Serialized 173 byte log entry
0112 02:52:35.223480 (+872186us) op_driver.cc:464] REPLICATION: finished
0112 02:52:35.301697 (+ 78217us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 02:52:35.334039 (+ 32342us) log.cc:844] Serialized 5 byte log entry
0112 02:52:35.467151 (+133112us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.474932 (+ 7781us) log.cc:844] Serialized 14 byte log entry
0112 02:52:35.474964 (+ 32us) alter_schema_op.cc:73] Released schema lock
0112 02:52:35.474968 (+ 4us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":78132,"prepare.queue_time_us":201008,"prepare.run_cpu_time_us":10166,"prepare.run_wall_time_us":182457,"raft.queue_time_us":61,"raft.run_cpu_time_us":467,"raft.run_wall_time_us":465,"replication_time_us":872273,"thread_start_us":312330,"threads_started":2,"wal-append.queue_time_us":176614}]]}
I20250112 02:52:35.516803 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbe30f9860e2492b90e8cedf8ff9daba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.519124 751 tablet_service.cc:1467] Processing CreateTablet for tablet a1123077650c4e89bffddd6d1264e428 (DEFAULT_TABLE table=default.Table_36b06d2edb8e4d23a3cb7021a8267db4 [id=3506f201d2674e38a353c87e1b2f26a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.519896 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99407ead9cf547e3bbe78c0d0b327e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.521958 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fea8f6f13d54945934ee3880e00c5e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.522183 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56ddc025a8314f6f84d229191eb56692. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.531845 2024 raft_consensus.cc:2238] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:35.532383 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37d38f5e525440f8a0f7822620933435. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.539340 1101 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:35.538291 2036 ts_tablet_manager.cc:1918] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:35.542747 2036 log.cc:1198] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/e5c10fad790046a88fbc8ebe0303c481
I20250112 02:52:35.549700 2185 tablet.cc:1721] T abfaad67d5e34c0a8338dccc0ef05466 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:35.551007 2094 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:49624 (request call id 77) took 1301 ms. Trace:
I20250112 02:52:35.552021 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5524d75599f04becbe6a94426adb7026. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.548367 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:35.556680 757 tablet_service.cc:1467] Processing CreateTablet for tablet f559dedfb7854cf890bef362c50b4213 (DEFAULT_TABLE table=default.Table_7921b99e24f84cceb58128f18925ad49 [id=0525f87eccbb4a1d8726d18a00f9a4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.495440 2172 rpcz_store.cc:276] 0112 02:52:34.087257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.087685 (+ 428us) service_pool.cc:225] Handling call
0112 02:52:35.460726 (+1373041us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:34.293326 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 02:52:34.309515 (+ 16189us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 02:52:34.309522 (+ 7us) alter_schema_op.cc:66] Acquired schema lock
0112 02:52:34.309526 (+ 4us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 02:52:34.310743 (+ 1217us) alter_schema_op.cc:128] START. Timestamp: 1736650354310447
0112 02:52:34.310751 (+ 8us) op_driver.cc:348] REPLICATION: starting
0112 02:52:34.370013 (+ 59262us) log.cc:844] Serialized 173 byte log entry
0112 02:52:34.888455 (+518442us) op_driver.cc:464] REPLICATION: finished
0112 02:52:35.029733 (+141278us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 02:52:35.278839 (+249106us) log.cc:844] Serialized 5 byte log entry
0112 02:52:35.452779 (+173940us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.460657 (+ 7878us) log.cc:844] Serialized 14 byte log entry
0112 02:52:35.460696 (+ 39us) alter_schema_op.cc:73] Released schema lock
0112 02:52:35.460699 (+ 3us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":141189,"mutex_wait_us":385,"prepare.queue_time_us":140334,"prepare.run_cpu_time_us":8154,"prepare.run_wall_time_us":331320,"raft.queue_time_us":1042,"raft.run_cpu_time_us":407,"raft.run_wall_time_us":405,"replication_time_us":577671,"thread_start_us":420751,"threads_started":2,"wal-append.queue_time_us":224274}]]}
I20250112 02:52:35.558516 2117 raft_consensus.cc:491] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:35.558624 2117 raft_consensus.cc:513] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.559108 2117 leader_election.cc:290] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:35.557185 2036 ts_tablet_manager.cc:1939] T e5c10fad790046a88fbc8ebe0303c481 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:35.562489 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1123077650c4e89bffddd6d1264e428. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.561944 826 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:35.559873 754 tablet_service.cc:1467] Processing CreateTablet for tablet 2796289c7e8f4b45b26508c50b57c640 (DEFAULT_TABLE table=default.Table_23c987a24fed4ceeb90fd25f0367b08f [id=a9f367de65284002a6a8ea94fc8cb7d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.562041 760 tablet_service.cc:1467] Processing CreateTablet for tablet 37d38f5e525440f8a0f7822620933435 (DEFAULT_TABLE table=default.Table_39f67a35c5c640dd8cbbcea89df5e4fc [id=9d267774bf59428e854651956fbd79be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.568431 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f559dedfb7854cf890bef362c50b4213. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.559927 2094 rpcz_store.cc:276] 0112 02:52:34.249719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.249762 (+ 43us) service_pool.cc:225] Handling call
0112 02:52:35.550963 (+1301201us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:34.481839 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 02:52:34.482048 (+ 209us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 02:52:34.482059 (+ 11us) alter_schema_op.cc:66] Acquired schema lock
0112 02:52:34.482068 (+ 9us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 02:52:34.482108 (+ 40us) alter_schema_op.cc:128] START. Timestamp: 1736650354482020
0112 02:52:34.482113 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:34.482349 (+ 236us) log.cc:844] Serialized 173 byte log entry
0112 02:52:35.237248 (+754899us) op_driver.cc:464] REPLICATION: finished
0112 02:52:35.504303 (+267055us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 02:52:35.523195 (+ 18892us) log.cc:844] Serialized 5 byte log entry
0112 02:52:35.550523 (+ 27328us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.550882 (+ 359us) log.cc:844] Serialized 14 byte log entry
0112 02:52:35.550927 (+ 45us) alter_schema_op.cc:73] Released schema lock
0112 02:52:35.550932 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":266989,"prepare.queue_time_us":231757,"prepare.run_cpu_time_us":6954,"prepare.run_wall_time_us":153071,"raft.queue_time_us":4858,"raft.run_cpu_time_us":484,"raft.run_wall_time_us":6178,"replication_time_us":755097,"thread_start_us":413749,"threads_started":2,"wal-append.queue_time_us":143750}]]}
I20250112 02:52:35.570566 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37d38f5e525440f8a0f7822620933435. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.576192 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42eed0a9f8354e5391ec2463dafdb683. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.577890 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1811ce8982fd4003a35e89ea34e32746" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:35.578953 855 leader_election.cc:343] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:35.580796 759 tablet_service.cc:1467] Processing CreateTablet for tablet 430a8652c53e4801a1c72de529c2adcb (DEFAULT_TABLE table=default.Table_b645d30cd43d4f09a3ac34db205a80f9 [id=aa437a13d9bc4b05a2f99a6e494ef36a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.582196 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcb7e0181cf74bbea6036c106516a7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.583478 898 tablet_service.cc:1467] Processing CreateTablet for tablet 430a8652c53e4801a1c72de529c2adcb (DEFAULT_TABLE table=default.Table_b645d30cd43d4f09a3ac34db205a80f9 [id=aa437a13d9bc4b05a2f99a6e494ef36a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.583943 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 430a8652c53e4801a1c72de529c2adcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.587821 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f52b838214c54ac69ad30dac6e56669a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:35.593364 714 consensus_peers.cc:487] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 161fe26b9c0e4543b7aeb0021506d72a. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:35.594908 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2796289c7e8f4b45b26508c50b57c640. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:35.597517 714 consensus_peers.cc:487] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9199b0d2b924a79938a979df4196bb4. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:35.599078 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 430a8652c53e4801a1c72de529c2adcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.600113 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea33053c62dc4707b759d5c1756ab4bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.602614 2117 raft_consensus.cc:491] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:35.602763 2117 raft_consensus.cc:513] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.604673 2117 leader_election.cc:290] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:35.572253 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:35.609309 1045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d42f1611de49d6b6e5aaa9be907054" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:35.609472 2207 raft_consensus.cc:491] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:35.609869 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 2c03b5e612d64314b210c11686fdfa2a (table default.Table_4934f6022f2441eda88751236a5798be [id=5c4769bfcefd4fe0a0f3c2e315d50c32]). Expected version 6 got 0
I20250112 02:52:35.610062 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 4916ae3917084cfe8c93240a8857be0e (table default.Table_69f363c4da234792886b544823b3214e [id=c7052afaf6284ef0972835c1f46cb829]). Expected version 1 got 0
I20250112 02:52:35.610435 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:35.614003 2024 raft_consensus.cc:2267] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:35.616091 2207 raft_consensus.cc:3054] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:35.619228 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:35.621206 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c69587c5a09d40f0baba5c2f6e8a7b41 (table default.Table_1216acc1fa6b4e51a55a89ffe22bcc9a [id=b5457e4343ee43618c8309e737b69ec3]). Expected version 1 got 0
I20250112 02:52:35.622948 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:35.623996 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 161fe26b9c0e4543b7aeb0021506d72a (table default.Table_981a70d4d3184fd8a99e6b81ae1caab7 [id=8016eb490f574ad8a5d3ee598fecaeb5]) successfully deleted
I20250112 02:52:35.624603 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ee98f055024429eafaa29bb961cab87" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:35.624781 918 raft_consensus.cc:2463] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:35.625149 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet b9199b0d2b924a79938a979df4196bb4 (table default.Table_44769a7c8b7e463dbe5d907dacc0410f [id=472b3381fca44299922af20255b6aad1]) successfully deleted
W20250112 02:52:35.625402 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 416fb6820cd54e118950c68d381f51b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 416fb6820cd54e118950c68d381f51b2 already in progress: creating tablet
I20250112 02:52:35.627710 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 43) took 1423 ms. Trace:
I20250112 02:52:35.627863 2036 rpcz_store.cc:276] 0112 02:52:34.204101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.276429 (+ 72328us) service_pool.cc:225] Handling call
0112 02:52:35.160254 (+883825us) ts_tablet_manager.cc:1145] Deleting tablet e5c10fad790046a88fbc8ebe0303c481
0112 02:52:35.160262 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:35.205942 (+ 45680us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.538139 (+332197us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.627697 (+ 89558us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":883717}
I20250112 02:52:35.628576 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c1ae3f34b434ee5bc3519cce8d283a6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:35.628675 1027 consensus_meta.cc:287] T 2d47f6d663934b25a0200ee9fa81165d P f1e775296eb746db8b318f17d26820e6: Time spent flushing consensus metadata: real 0.535s user 0.002s sys 0.002s
I20250112 02:52:35.629441 2036 tablet_replica.cc:331] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:35.629535 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet e5c10fad790046a88fbc8ebe0303c481 (table default.Table_16b036d8833a47a99577ab67579ccb5a [id=eb980088c4584ccf82bb585c2210894a]) successfully deleted
I20250112 02:52:35.629702 2024 ts_tablet_manager.cc:1905] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:35.629724 2036 raft_consensus.cc:2238] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:35.629933 2036 raft_consensus.cc:2267] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:35.630443 2036 ts_tablet_manager.cc:1905] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:35.630771 2218 raft_consensus.cc:491] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 436220c23fa948efb5b6ca15875d8b0e)
I20250112 02:52:35.630962 2218 raft_consensus.cc:513] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:35.632007 464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd25f03f25df49178a5c53c636ecf304" candidate_term: 5 candidate_status { last_received { term: 4 index: 598 } } ignore_live_leader: false dest_uuid: "436220c23fa948efb5b6ca15875d8b0e" is_pre_election: true
I20250112 02:52:35.634159 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_5d473cabff2e46e0b531fbce28d9d558 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 1 got 0
I20250112 02:52:35.635475 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 23e7a2e88c374e949ced92c557225cbf (table default.Table_cd1f185d82b3474083246ec3702851fc [id=bfa9a504a8d34c86929efb0d15d049d0]). Expected version 3 got 0
W20250112 02:52:35.636463 854 leader_election.cc:343] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:35.636627 854 leader_election.cc:304] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:35.637084 2223 raft_consensus.cc:2743] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:35.627151 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7320ac55dbfa47fe851fff94d32e1ffe" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:35.637467 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:35.635512 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1811ce8982fd4003a35e89ea34e32746" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:35.638985 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:35.644580 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9c006281dc949119951c74e23167d7f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:35.644812 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9c006281dc949119951c74e23167d7f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:35.645747 855 leader_election.cc:343] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:35.643492 1173 raft_consensus.cc:357] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.646591 1173 raft_consensus.cc:383] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:35.646692 1173 raft_consensus.cc:738] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:35.646984 1173 consensus_queue.cc:260] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.647714 1173 ts_tablet_manager.cc:1428] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.399s user 0.002s sys 0.002s
I20250112 02:52:35.650028 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fe57d374eb44c3fb1532484e214b654. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.651024 742 tablet_service.cc:1467] Processing CreateTablet for tablet acafc4f579534a69a8f307ca4cf72e12 (DEFAULT_TABLE table=default.Table_cc762fcab6cf493ab58b3d1defecf106 [id=1b44f2cf98e246b0b3afbd1e3fa9676c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.668893 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acafc4f579534a69a8f307ca4cf72e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.653086 2207 raft_consensus.cc:513] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.659116 2216 tablet.cc:1721] T 7ea075826b16475ca474e284fc6bc545 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:35.674603 2185 tablet.cc:1721] T c69587c5a09d40f0baba5c2f6e8a7b41 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:35.676326 1173 tablet_bootstrap.cc:492] T 5cc414d6420c41839621ab858067548b P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:35.677150 2207 leader_election.cc:290] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:35.698053 2218 leader_election.cc:290] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 436220c23fa948efb5b6ca15875d8b0e (127.0.104.62:42341), f4a49a5756b143fcb767c3f41463f8d9 (127.0.104.61:38771)
I20250112 02:52:35.698590 2216 tablet.cc:1721] T d8eb0efc5d7f4c72bdfc7531a444d42a P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:35.706847 2222 raft_consensus.cc:491] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:35.707064 1727 raft_consensus.cc:695] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:35.706988 2222 raft_consensus.cc:513] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.707741 2222 leader_election.cc:290] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:35.708626 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003a3b4705e943c6a900762ec11ddd47" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:35.709242 855 leader_election.cc:343] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:35.712041 1727 consensus_queue.cc:237] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:35.714383 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 214954us
I20250112 02:52:35.716626 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 6117dc8b32a64535a2f4167a03bed0d2 (table default.Table_365739770a11434dbf260ea912e7ea81 [id=868adb627654450c936ef198e49d853b]). Expected version 1 got 0
I20250112 02:52:35.717031 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:35.717424 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 7ea075826b16475ca474e284fc6bc545 (table default.Table_009a6976f6db4f218e3c3373368a58d7 [id=c5005669bb5f4c029d673bd39cc1535d]). Expected version 1 got 0
I20250112 02:52:35.717953 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet f8c01b80fc8242d98fdfbf767212144d (table default.Table_63649e9607394f4bb72a3dd6e6d0dd69 [id=9807191b6e5c4d478cd47a54464fb9ce]). Expected version 1 got 0
I20250112 02:52:35.718650 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet f31b777e7e0f43a28839d8c48c6a6590 (table default.Table_5fb592e41cc4406aa27546e7b41fdb88 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 1 got 0
I20250112 02:52:35.718914 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:35.719154 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:35.720038 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:35.720311 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:35.721347 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 7cd9985cd20148f195a857c478b0755f (table default.Table_9b65dd291c9f4fcdaa6d5e37f04d24ae [id=e63daf724d874c89badafb8d59653f74]). Expected version 1 got 0
I20250112 02:52:35.721659 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:35.721948 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
I20250112 02:52:35.722195 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:35.722698 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:35.723049 453 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
W20250112 02:52:35.723451 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24ebf85f76f44633a20d97964638c0ac consensus state has no leader
I20250112 02:52:35.723642 449 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 113) took 1283 ms. Trace:
I20250112 02:52:35.723755 449 rpcz_store.cc:276] 0112 02:52:34.439723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.563787 (+124064us) service_pool.cc:225] Handling call
0112 02:52:34.564190 (+ 403us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:34.564387 (+ 197us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:34.564540 (+ 153us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:34.564608 (+ 68us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:34.586703 (+ 22095us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:35.723595 (+1136892us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:34.565025 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:34.565108 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:34.565114 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:34.565116 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:34.565312 (+ 196us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:34.565320 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:34.565324 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:34.565368 (+ 44us) tablet.cc:698] Row locks acquired
0112 02:52:34.565370 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:34.565393 (+ 23us) write_op.cc:270] Start()
0112 02:52:34.565414 (+ 21us) write_op.cc:276] Timestamp: P: 1736650354565369 usec, L: 0
0112 02:52:34.565417 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:34.565552 (+ 135us) log.cc:844] Serialized 405 byte log entry
0112 02:52:34.584404 (+ 18852us) op_driver.cc:464] REPLICATION: finished
0112 02:52:34.584954 (+ 550us) write_op.cc:301] APPLY: starting
0112 02:52:34.584963 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:34.584987 (+ 24us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:34.585009 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:34.585035 (+ 26us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:34.585055 (+ 20us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:34.585061 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:34.585110 (+ 49us) log.cc:844] Serialized 23 byte log entry
0112 02:52:34.585138 (+ 28us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:34.585147 (+ 9us) write_op.cc:454] Released schema lock
0112 02:52:34.585151 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:34.585196 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:34.585199 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":524,"apply.run_cpu_time_us":238,"apply.run_wall_time_us":237,"num_ops":1,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":502,"prepare.run_wall_time_us":654,"raft.queue_time_us":36,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":18974}]]}
I20250112 02:52:35.724797 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_c586400629b14725a93d61c2c6e4ad82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:35.725412 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c586400629b14725a93d61c2c6e4ad82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:35.727945 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 79eb61cedd544c49869471eebe2fd237 consensus state has no leader
I20250112 02:52:35.728121 455 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 111) took 1288 ms. Trace:
I20250112 02:52:35.728217 455 rpcz_store.cc:276] 0112 02:52:34.439520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.466541 (+ 27021us) service_pool.cc:225] Handling call
0112 02:52:34.467724 (+ 1183us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:34.467928 (+ 204us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:34.468088 (+ 160us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:34.468250 (+ 162us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:34.492828 (+ 24578us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:35.728086 (+1235258us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:34.468637 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:34.468720 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:34.468726 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:34.468729 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:34.468809 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:34.468818 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:34.468822 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:34.468869 (+ 47us) tablet.cc:698] Row locks acquired
0112 02:52:34.468872 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:34.468908 (+ 36us) write_op.cc:270] Start()
0112 02:52:34.468933 (+ 25us) write_op.cc:276] Timestamp: P: 1736650354468794 usec, L: 0
0112 02:52:34.468937 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:34.469064 (+ 127us) log.cc:844] Serialized 405 byte log entry
0112 02:52:34.491572 (+ 22508us) op_driver.cc:464] REPLICATION: finished
0112 02:52:34.491697 (+ 125us) write_op.cc:301] APPLY: starting
0112 02:52:34.491709 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:34.491741 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:34.491780 (+ 39us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:34.491819 (+ 39us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:34.491855 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:34.491861 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:34.491931 (+ 70us) log.cc:844] Serialized 23 byte log entry
0112 02:52:34.491979 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:34.491996 (+ 17us) write_op.cc:454] Released schema lock
0112 02:52:34.492002 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:34.492061 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:34.492067 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":83,"apply.run_cpu_time_us":356,"apply.run_wall_time_us":356,"num_ops":1,"prepare.queue_time_us":40,"prepare.run_cpu_time_us":507,"prepare.run_wall_time_us":799,"raft.queue_time_us":29,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":22609}]]}
I20250112 02:52:35.729085 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_63abe3ac694a4ed6bf4aef720b542e73" } modify_external_catalogs: true
I20250112 02:52:35.729306 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_63abe3ac694a4ed6bf4aef720b542e73" } modify_external_catalogs: true
I20250112 02:52:35.730619 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 82b351c563d749df806ab5d0331a4ef2
I20250112 02:52:35.731542 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b351c563d749df806ab5d0331a4ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:35.735692 1177 tablet_bootstrap.cc:654] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:35.736292 454 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40610 (request call id 112) took 1296 ms. Trace:
I20250112 02:52:35.736536 2003 ts_tablet_manager.cc:1918] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:35.736411 454 rpcz_store.cc:276] 0112 02:52:34.439622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.556596 (+116974us) service_pool.cc:225] Handling call
0112 02:52:34.557031 (+ 435us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:34.557191 (+ 160us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:52:34.557259 (+ 68us) catalog_manager.cc:2762] Locking tablets
0112 02:52:34.557386 (+ 127us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:52:34.588037 (+ 30651us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:52:34.588132 (+ 95us) catalog_manager.cc:2810] Committing in-memory state
0112 02:52:35.730555 (+1142423us) catalog_manager.cc:2814] Aborting table tasks
0112 02:52:35.736248 (+ 5693us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:34.559538 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:34.559609 (+ 71us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:34.559614 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:34.559615 (+ 1us) tablet.cc:659] Decoding operations
0112 02:52:34.559707 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:34.559714 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:34.559718 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:34.559788 (+ 70us) tablet.cc:698] Row locks acquired
0112 02:52:34.559791 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:34.559817 (+ 26us) write_op.cc:270] Start()
0112 02:52:34.559841 (+ 24us) write_op.cc:276] Timestamp: P: 1736650354559791 usec, L: 0
0112 02:52:34.559846 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:34.559968 (+ 122us) log.cc:844] Serialized 756 byte log entry
0112 02:52:34.584349 (+ 24381us) op_driver.cc:464] REPLICATION: finished
0112 02:52:34.584533 (+ 184us) write_op.cc:301] APPLY: starting
0112 02:52:34.584544 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:34.584577 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:34.584609 (+ 32us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:34.584665 (+ 56us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:34.584691 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:34.584697 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:34.584755 (+ 58us) log.cc:844] Serialized 29 byte log entry
0112 02:52:34.584802 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:34.584816 (+ 14us) write_op.cc:454] Released schema lock
0112 02:52:34.584822 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:34.584877 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:34.584880 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":148,"apply.run_cpu_time_us":336,"apply.run_wall_time_us":336,"num_ops":2,"prepare.queue_time_us":1665,"prepare.run_cpu_time_us":518,"prepare.run_wall_time_us":852,"raft.queue_time_us":34,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":24474}]]}
I20250112 02:52:35.737315 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_75f163a2f4aa48df95c9bafd3b311f4b" } modify_external_catalogs: true
I20250112 02:52:35.737475 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_75f163a2f4aa48df95c9bafd3b311f4b" } modify_external_catalogs: true
I20250112 02:52:35.739153 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b351c563d749df806ab5d0331a4ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:35.740674 450 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 104) took 1317 ms. Trace:
I20250112 02:52:35.740772 450 rpcz_store.cc:276] 0112 02:52:34.423206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.429526 (+ 6320us) service_pool.cc:225] Handling call
0112 02:52:34.429872 (+ 346us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:34.430079 (+ 207us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:34.430224 (+ 145us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:34.430278 (+ 54us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:34.483181 (+ 52903us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:35.740618 (+1257437us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:34.435053 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:34.435139 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:34.435152 (+ 13us) write_op.cc:435] Acquired schema lock
0112 02:52:34.435154 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:34.435231 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:34.435238 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:34.435242 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:34.435290 (+ 48us) tablet.cc:698] Row locks acquired
0112 02:52:34.435292 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:34.435316 (+ 24us) write_op.cc:270] Start()
0112 02:52:34.435338 (+ 22us) write_op.cc:276] Timestamp: P: 1736650354435219 usec, L: 0
0112 02:52:34.435341 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:34.435509 (+ 168us) log.cc:844] Serialized 405 byte log entry
0112 02:52:34.442749 (+ 7240us) op_driver.cc:464] REPLICATION: finished
0112 02:52:34.443552 (+ 803us) write_op.cc:301] APPLY: starting
0112 02:52:34.443561 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:34.443587 (+ 26us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:34.443617 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:34.443652 (+ 35us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:34.443676 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:34.443682 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:34.443733 (+ 51us) log.cc:844] Serialized 23 byte log entry
0112 02:52:34.443787 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:34.443798 (+ 11us) write_op.cc:454] Released schema lock
0112 02:52:34.443802 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:34.443853 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:34.443856 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":773,"apply.run_cpu_time_us":296,"apply.run_wall_time_us":296,"num_ops":1,"prepare.queue_time_us":4362,"prepare.run_cpu_time_us":509,"prepare.run_wall_time_us":509,"replication_time_us":7395}]]}
I20250112 02:52:35.742282 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_fc264b6a06584c7ea757f5d18e05e2bc" } modify_external_catalogs: true
I20250112 02:52:35.742426 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_fc264b6a06584c7ea757f5d18e05e2bc" } modify_external_catalogs: true
I20250112 02:52:35.743978 447 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 110) took 1304 ms. Trace:
I20250112 02:52:35.744124 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 003a3b4705e943c6a900762ec11ddd47
I20250112 02:52:35.743988 452 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 126) took 1061 ms. Trace:
I20250112 02:52:35.744634 446 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40610 (request call id 98) took 1421 ms. Trace:
I20250112 02:52:35.744814 2003 log.cc:1198] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/1c639a2bcd4740e4be7b2b162004669a
I20250112 02:52:35.745003 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 003a3b4705e943c6a900762ec11ddd47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:35.744724 446 rpcz_store.cc:276] 0112 02:52:34.322964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.379048 (+ 56084us) service_pool.cc:225] Handling call
0112 02:52:34.380154 (+ 1106us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:34.380292 (+ 138us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:52:34.380345 (+ 53us) catalog_manager.cc:2762] Locking tablets
0112 02:52:34.847976 (+467631us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:52:34.855658 (+ 7682us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:52:34.855762 (+ 104us) catalog_manager.cc:2810] Committing in-memory state
0112 02:52:35.744043 (+888281us) catalog_manager.cc:2814] Aborting table tasks
0112 02:52:35.744582 (+ 539us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:34.848903 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:34.849014 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:34.849022 (+ 8us) write_op.cc:435] Acquired schema lock
0112 02:52:34.849025 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:34.849159 (+ 134us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:34.849175 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:34.849182 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:34.849332 (+ 150us) tablet.cc:698] Row locks acquired
0112 02:52:34.849337 (+ 5us) write_op.cc:260] PREPARE: finished
0112 02:52:34.849377 (+ 40us) write_op.cc:270] Start()
0112 02:52:34.849416 (+ 39us) write_op.cc:276] Timestamp: P: 1736650354849291 usec, L: 0
0112 02:52:34.849421 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:34.849632 (+ 211us) log.cc:844] Serialized 756 byte log entry
0112 02:52:34.854127 (+ 4495us) op_driver.cc:464] REPLICATION: finished
0112 02:52:34.854258 (+ 131us) write_op.cc:301] APPLY: starting
0112 02:52:34.854270 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:34.854310 (+ 40us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:34.854337 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:34.854398 (+ 61us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:34.854428 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:34.854435 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:34.854513 (+ 78us) log.cc:844] Serialized 29 byte log entry
0112 02:52:34.854563 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:34.854581 (+ 18us) write_op.cc:454] Released schema lock
0112 02:52:34.854587 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:34.854659 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:34.854663 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":91,"apply.run_cpu_time_us":394,"apply.run_wall_time_us":394,"num_ops":2,"prepare.queue_time_us":68,"prepare.run_cpu_time_us":842,"prepare.run_wall_time_us":842,"raft.queue_time_us":1255,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":291,"replication_time_us":4681}]]}
I20250112 02:52:35.745246 2003 ts_tablet_manager.cc:1939] T 1c639a2bcd4740e4be7b2b162004669a P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:35.745510 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_cb4ed576320c497588e65dcbac70afe9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:35.744123 447 rpcz_store.cc:276] 0112 02:52:34.439388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.460826 (+ 21438us) service_pool.cc:225] Handling call
0112 02:52:34.461450 (+ 624us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:34.461645 (+ 195us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:34.461930 (+ 285us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:34.462002 (+ 72us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:34.481655 (+ 19653us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:35.743931 (+1262276us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:34.462470 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:34.462543 (+ 73us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:34.462548 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:34.462550 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:34.462631 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:34.462642 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:34.462646 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:34.462695 (+ 49us) tablet.cc:698] Row locks acquired
0112 02:52:34.462698 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:34.462730 (+ 32us) write_op.cc:270] Start()
0112 02:52:34.462753 (+ 23us) write_op.cc:276] Timestamp: P: 1736650354462627 usec, L: 0
0112 02:52:34.462757 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:34.462932 (+ 175us) log.cc:844] Serialized 405 byte log entry
0112 02:52:34.476737 (+ 13805us) op_driver.cc:464] REPLICATION: finished
0112 02:52:34.476852 (+ 115us) write_op.cc:301] APPLY: starting
0112 02:52:34.476863 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:34.476917 (+ 54us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:34.476938 (+ 21us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:34.476974 (+ 36us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:34.477016 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:34.477023 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:34.477079 (+ 56us) log.cc:844] Serialized 23 byte log entry
0112 02:52:34.477124 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:34.477137 (+ 13us) write_op.cc:454] Released schema lock
0112 02:52:34.477143 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:34.477198 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:34.477202 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":76,"apply.run_cpu_time_us":338,"apply.run_wall_time_us":337,"num_ops":1,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":559,"prepare.run_wall_time_us":769,"raft.queue_time_us":243,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":140,"replication_time_us":13956}]]}
W20250112 02:52:35.745999 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb4ed576320c497588e65dcbac70afe9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:35.745995 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 28) took 1669 ms. Trace:
I20250112 02:52:35.746097 2003 rpcz_store.cc:276] 0112 02:52:34.076808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.076945 (+ 137us) service_pool.cc:225] Handling call
0112 02:52:35.243489 (+1166544us) ts_tablet_manager.cc:1145] Deleting tablet 1c639a2bcd4740e4be7b2b162004669a
0112 02:52:35.243495 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:35.700673 (+457178us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.718166 (+ 17493us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.745962 (+ 27796us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":105578,"tablet-delete.queue_time_us":1146647}
I20250112 02:52:35.746227 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_635bfb7ffd504fb89c9340cdf227944d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:35.746487 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 30) took 1659 ms. Trace:
I20250112 02:52:35.746583 2003 rpcz_store.cc:276] 0112 02:52:34.087041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.087776 (+ 735us) service_pool.cc:225] Handling call
0112 02:52:35.746361 (+1658585us) ts_tablet_manager.cc:1145] Deleting tablet 87759e8b99b04e628badc8f60cb38bb3
0112 02:52:35.746370 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:35.746460 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1649361}
W20250112 02:52:35.746676 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_635bfb7ffd504fb89c9340cdf227944d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:35.746886 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 37) took 1650 ms. Trace:
I20250112 02:52:35.746978 2003 rpcz_store.cc:276] 0112 02:52:34.096227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.096289 (+ 62us) service_pool.cc:225] Handling call
0112 02:52:35.746780 (+1650491us) ts_tablet_manager.cc:1145] Deleting tablet fe614eaeb793473d9b2da206b7345562
0112 02:52:35.746789 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:35.746862 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1649684}
I20250112 02:52:35.747264 2003 tablet_replica.cc:331] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:35.744369 452 rpcz_store.cc:276] 0112 02:52:34.682274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.978981 (+296707us) service_pool.cc:225] Handling call
0112 02:52:34.982596 (+ 3615us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:34.982822 (+ 226us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:34.982995 (+ 173us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:34.983093 (+ 98us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:34.993480 (+ 10387us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:34.993486 (+ 6us) write_op.cc:454] Released schema lock
0112 02:52:34.993546 (+ 60us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:35.743931 (+750385us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:34.987162 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:34.987261 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:34.987267 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:34.987282 (+ 15us) tablet.cc:659] Decoding operations
0112 02:52:34.987375 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:34.987386 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:34.987392 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:34.987452 (+ 60us) tablet.cc:698] Row locks acquired
0112 02:52:34.987455 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:34.987510 (+ 55us) write_op.cc:270] Start()
0112 02:52:34.987541 (+ 31us) write_op.cc:276] Timestamp: P: 1736650354987397 usec, L: 0
0112 02:52:34.987544 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:34.987748 (+ 204us) log.cc:844] Serialized 405 byte log entry
0112 02:52:34.990869 (+ 3121us) op_driver.cc:464] REPLICATION: finished
0112 02:52:34.991018 (+ 149us) write_op.cc:301] APPLY: starting
0112 02:52:34.991034 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:34.991079 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:34.991120 (+ 41us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:34.991184 (+ 64us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:34.991227 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:34.991239 (+ 12us) write_op.cc:312] APPLY: finished
0112 02:52:34.991334 (+ 95us) log.cc:844] Serialized 23 byte log entry
0112 02:52:34.991399 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:34.991421 (+ 22us) write_op.cc:454] Released schema lock
0112 02:52:34.991429 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":1,"child_traces":[["op",{"apply.queue_time_us":102,"apply.run_cpu_time_us":496,"apply.run_wall_time_us":1021,"num_ops":1,"prepare.queue_time_us":194,"prepare.run_cpu_time_us":688,"prepare.run_wall_time_us":688,"raft.queue_time_us":988,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":212,"replication_time_us":3296}]]}
I20250112 02:52:35.748317 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_78163902681c4b26885de83c218c408f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:35.748807 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78163902681c4b26885de83c218c408f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:35.751116 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 1c639a2bcd4740e4be7b2b162004669a (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]) successfully deleted
W20250112 02:52:35.751406 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 87759e8b99b04e628badc8f60cb38bb3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87759e8b99b04e628badc8f60cb38bb3 already in progress: creating tablet
W20250112 02:52:35.751719 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet fe614eaeb793473d9b2da206b7345562 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe614eaeb793473d9b2da206b7345562 already in progress: creating tablet
I20250112 02:52:35.760331 1177 tablet_bootstrap.cc:492] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:35.748654 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 003a3b4705e943c6a900762ec11ddd47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:35.747455 2003 raft_consensus.cc:2238] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:35.764485 2003 raft_consensus.cc:2267] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:35.765182 2003 ts_tablet_manager.cc:1905] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:35.764051 1177 ts_tablet_manager.cc:1397] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.354s user 0.004s sys 0.006s
I20250112 02:52:35.769631 1177 raft_consensus.cc:357] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.770542 1177 raft_consensus.cc:383] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:35.770653 1177 raft_consensus.cc:738] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:35.770942 1177 consensus_queue.cc:260] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.773531 1177 ts_tablet_manager.cc:1428] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250112 02:52:35.773954 2160 tablet.cc:1721] T f8c01b80fc8242d98fdfbf767212144d P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:35.774362 2159 tablet.cc:1721] T f8c01b80fc8242d98fdfbf767212144d P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:35.774519 1177 tablet_bootstrap.cc:492] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:35.776134 1177 tablet_bootstrap.cc:654] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:35.780156 899 tablet_service.cc:1467] Processing CreateTablet for tablet acafc4f579534a69a8f307ca4cf72e12 (DEFAULT_TABLE table=default.Table_cc762fcab6cf493ab58b3d1defecf106 [id=1b44f2cf98e246b0b3afbd1e3fa9676c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.780777 2214 tablet.cc:1721] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:35.781045 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b70dce4d1d4e29997a18297ac0f82a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:35.781392 742 tablet_service.cc:1467] Processing CreateTablet for tablet a4328137407843e1879493ef0c56ffc8 (DEFAULT_TABLE table=default.Table_c3e55b9696194ba58cbc9f1bb391b27d [id=9049d21b5e854418b070560cecda2eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.781494 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763f77385e404c149c16edabe469c299" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:35.781538 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a34d03a80ae402db6607261a5c40b60" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:35.781738 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4328137407843e1879493ef0c56ffc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.781868 2024 ts_tablet_manager.cc:1918] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:35.781988 2024 log.cc:1198] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/cb724feccc7f4fbc846fb6fa61be3385
I20250112 02:52:35.782366 2024 ts_tablet_manager.cc:1939] T cb724feccc7f4fbc846fb6fa61be3385 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:35.782649 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 36) took 1368 ms. Trace:
I20250112 02:52:35.783239 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acafc4f579534a69a8f307ca4cf72e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.782770 2024 rpcz_store.cc:276] 0112 02:52:34.414654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.415390 (+ 736us) service_pool.cc:225] Handling call
0112 02:52:35.245613 (+830223us) ts_tablet_manager.cc:1145] Deleting tablet cb724feccc7f4fbc846fb6fa61be3385
0112 02:52:35.245622 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:35.454688 (+209066us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x616000142730. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8cac6d90 00007f2e8cac5342 00007f2e8f569eb7 00007f2e8f59c584 00007f2e8f57d690 00007f2e8f57d41d 0000000000a71ecc 00007f2e7ff74b64 00007f2e7ff7818b 00007f2e7ff78008
0112 02:52:35.668105 (+213417us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.778941 (+110836us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.782618 (+ 3677us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4235,"spinlock_wait_cycles":53232896,"tablet-delete.queue_time_us":828997}
I20250112 02:52:35.789878 2024 tablet_replica.cc:331] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:35.790131 2024 raft_consensus.cc:2238] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
W20250112 02:52:35.797251 1345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20250112 02:52:35.804149 2159 tablet.cc:1721] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:35.804569 2160 tablet.cc:1721] T 3f5b418c17084420b867498fda79fb0a P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:35.809033 899 tablet_service.cc:1467] Processing CreateTablet for tablet a4328137407843e1879493ef0c56ffc8 (DEFAULT_TABLE table=default.Table_c3e55b9696194ba58cbc9f1bb391b27d [id=9049d21b5e854418b070560cecda2eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.809698 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4328137407843e1879493ef0c56ffc8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:35.809813 1181 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.040s sys 0.108s
W20250112 02:52:35.809877 1181 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.575s user 0.040s sys 0.108s
I20250112 02:52:35.817354 1177 tablet_bootstrap.cc:492] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:35.817643 1177 ts_tablet_manager.cc:1397] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.003s
W20250112 02:52:35.817811 991 leader_election.cc:343] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:35.817991 991 leader_election.cc:304] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:35.818620 1177 raft_consensus.cc:357] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.818871 1177 raft_consensus.cc:383] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:35.818976 1177 raft_consensus.cc:738] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:35.819301 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73136be25c79491784d3ea71aa9b27f2" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:35.819999 1181 tablet_bootstrap.cc:492] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:35.820155 1181 ts_tablet_manager.cc:1397] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.587s user 0.041s sys 0.108s
I20250112 02:52:35.820849 1181 raft_consensus.cc:357] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.821077 1181 raft_consensus.cc:383] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:35.821175 1181 raft_consensus.cc:738] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:35.821494 1181 consensus_queue.cc:260] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.822041 1181 ts_tablet_manager.cc:1428] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:35.816478 2208 consensus_queue.cc:1035] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20250112 02:52:35.822497 1181 tablet_bootstrap.cc:492] T 781fe3514f4c4f80b1ee49278502ccc1 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:35.825852 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 29d7774d4a284d24b833d50809996fc2 (DEFAULT_TABLE table=default.Table_84016330aad94c2ba3aec196a69d28de [id=84d89da1bbaf4f5985335bb97a9f6d86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.826195 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29d7774d4a284d24b833d50809996fc2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:35.826236 1345 leader_election.cc:336] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f4a49a5756b143fcb767c3f41463f8d9 (127.0.104.61:38771): Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection refused (error 111)
I20250112 02:52:35.826385 1345 leader_election.cc:304] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd25f03f25df49178a5c53c636ecf304; no voters: 436220c23fa948efb5b6ca15875d8b0e, f4a49a5756b143fcb767c3f41463f8d9
I20250112 02:52:35.826622 2218 raft_consensus.cc:2743] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250112 02:52:35.827042 1177 consensus_queue.cc:260] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.828866 889 tablet_service.cc:1467] Processing CreateTablet for tablet 940affdc2c484b7d9fd2de17f9d1cea7 (DEFAULT_TABLE table=default.Table_9afc6941825a45bc8f98bf373ff5f27c [id=c76eb62dc8bf4b97b675787374189057]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.829047 1177 ts_tablet_manager.cc:1428] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.011s user 0.001s sys 0.002s
I20250112 02:52:35.829037 2056 consensus_queue.cc:1035] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20250112 02:52:35.829190 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 940affdc2c484b7d9fd2de17f9d1cea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.831179 756 tablet_service.cc:1467] Processing CreateTablet for tablet 940affdc2c484b7d9fd2de17f9d1cea7 (DEFAULT_TABLE table=default.Table_9afc6941825a45bc8f98bf373ff5f27c [id=c76eb62dc8bf4b97b675787374189057]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.831817 1173 tablet_bootstrap.cc:654] T 5cc414d6420c41839621ab858067548b P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:35.832044 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe57d374eb44c3fb1532484e214b654 (DEFAULT_TABLE table=default.Table_6730101812ca4bca8e85617d952cc549 [id=45403aee8fed439c94e3b7d3c8898668]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.832880 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 0fea8f6f13d54945934ee3880e00c5e9 (DEFAULT_TABLE table=default.Table_f38c3f9c44b741219f03cf46cfbc9937 [id=a41ea1ae71784a99a0ca89a45825cb85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.834306 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 940affdc2c484b7d9fd2de17f9d1cea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.834282 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 56ddc025a8314f6f84d229191eb56692 (DEFAULT_TABLE table=default.Table_b2879d6a85294514989d7b68d3ac97d4 [id=2ddd5dd296b94bc8b96a97b66e775836]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.834699 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 5b741e4f6abd4695b3ba1d7a667b6046 (DEFAULT_TABLE table=default.Table_e781b72b17d54bf8ba293cd013e5e14d [id=332595f0d3a24adf8b9bbe47b82b9f66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.834959 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 7c1ae3f34b434ee5bc3519cce8d283a6 (DEFAULT_TABLE table=default.Table_b236e732be7844a08eaa268cb451d22f [id=48b14376b80643f5a6d8e58df085d8f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.835433 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fea8f6f13d54945934ee3880e00c5e9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:35.836206 991 leader_election.cc:343] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:35.836300 991 leader_election.cc:304] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:35.841949 1017 tablet_service.cc:1467] Processing CreateTablet for tablet f52b838214c54ac69ad30dac6e56669a (DEFAULT_TABLE table=default.Table_2989b0fb83514c45ac9836e95e444f31 [id=f612d417385a4a02814d993aab77b5a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.843545 2207 raft_consensus.cc:2743] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:35.844087 2215 raft_consensus.cc:2743] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:35.844261 1177 tablet_bootstrap.cc:492] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:35.844221 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 99407ead9cf547e3bbe78c0d0b327e6a (DEFAULT_TABLE table=default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.844506 1034 tablet_service.cc:1467] Processing CreateTablet for tablet cbe30f9860e2492b90e8cedf8ff9daba (DEFAULT_TABLE table=default.Table_75f163a2f4aa48df95c9bafd3b311f4b [id=e6546a8535524b5996680a4e5451032f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.845103 2056 consensus_queue.cc:1035] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20250112 02:52:35.846606 1177 tablet_bootstrap.cc:654] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:35.849097 742 tablet_service.cc:1467] Processing CreateTablet for tablet cd52261c4a414e7d93a528e2eddfa9db (DEFAULT_TABLE table=default.Table_70d1488de98d4380b8024db852f58991 [id=407e1e8894524b7682f498b3904b06e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.850054 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd52261c4a414e7d93a528e2eddfa9db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.852506 898 tablet_service.cc:1467] Processing CreateTablet for tablet cd52261c4a414e7d93a528e2eddfa9db (DEFAULT_TABLE table=default.Table_70d1488de98d4380b8024db852f58991 [id=407e1e8894524b7682f498b3904b06e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.852998 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd52261c4a414e7d93a528e2eddfa9db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:35.854981 1026 tablet_service.cc:1467] Processing CreateTablet for tablet a1123077650c4e89bffddd6d1264e428 (DEFAULT_TABLE table=default.Table_36b06d2edb8e4d23a3cb7021a8267db4 [id=3506f201d2674e38a353c87e1b2f26a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.855800 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 5524d75599f04becbe6a94426adb7026 (DEFAULT_TABLE table=default.Table_ef2001a2d9b24b4bb1f8fb9102703022 [id=b7739f28885a453d9ef3966511183c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.861644 2003 ts_tablet_manager.cc:1918] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:35.863080 2003 log.cc:1198] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/b9199b0d2b924a79938a979df4196bb4
I20250112 02:52:35.867017 2003 ts_tablet_manager.cc:1939] T b9199b0d2b924a79938a979df4196bb4 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:35.868067 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 34) took 1780 ms. Trace:
I20250112 02:52:35.868245 2003 rpcz_store.cc:276] 0112 02:52:34.087501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.087758 (+ 257us) service_pool.cc:225] Handling call
0112 02:52:35.747207 (+1659449us) ts_tablet_manager.cc:1145] Deleting tablet b9199b0d2b924a79938a979df4196bb4
0112 02:52:35.747215 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:35.819680 (+ 72465us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.861532 (+ 41852us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.868004 (+ 6472us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1650037}
W20250112 02:52:35.871389 1778 thread.cc:635] rpc reactor (reactor) Time spent creating pthread: real 0.517s user 0.072s sys 0.125s
W20250112 02:52:35.871531 1778 thread.cc:601] rpc reactor (reactor) Time spent starting thread: real 0.517s user 0.072s sys 0.125s
I20250112 02:52:35.872769 2003 tablet_replica.cc:331] T bb217afc08304c0ea35dee3120347ba1 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:35.872856 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet b9199b0d2b924a79938a979df4196bb4 (table default.Table_44769a7c8b7e463dbe5d907dacc0410f [id=472b3381fca44299922af20255b6aad1]) successfully deleted
I20250112 02:52:35.873042 2003 raft_consensus.cc:2238] T bb217afc08304c0ea35dee3120347ba1 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:35.873270 2003 raft_consensus.cc:2267] T bb217afc08304c0ea35dee3120347ba1 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:35.874709 2003 ts_tablet_manager.cc:1905] T bb217afc08304c0ea35dee3120347ba1 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:35.874179 1177 tablet_bootstrap.cc:492] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:35.875835 1177 ts_tablet_manager.cc:1397] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.001s
I20250112 02:52:35.876674 1177 raft_consensus.cc:357] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.876943 1177 raft_consensus.cc:383] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:35.877043 1177 raft_consensus.cc:738] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:35.877323 1177 consensus_queue.cc:260] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.877909 1177 ts_tablet_manager.cc:1428] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:35.878633 2036 ts_tablet_manager.cc:1918] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:35.878726 2036 log.cc:1198] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/5dd48d0e68014e1aa0c44620527baec9
I20250112 02:52:35.878996 2036 ts_tablet_manager.cc:1939] T 5dd48d0e68014e1aa0c44620527baec9 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:35.879356 1177 tablet_bootstrap.cc:492] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:35.879755 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 44) took 1675 ms. Trace:
I20250112 02:52:35.879837 2036 rpcz_store.cc:276] 0112 02:52:34.204521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.276546 (+ 72025us) service_pool.cc:225] Handling call
0112 02:52:35.629403 (+1352857us) ts_tablet_manager.cc:1145] Deleting tablet 5dd48d0e68014e1aa0c44620527baec9
0112 02:52:35.629411 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:35.634155 (+ 4744us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.878555 (+244400us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.879738 (+ 1183us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1352746}
I20250112 02:52:35.880198 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 5dd48d0e68014e1aa0c44620527baec9 (table default.Table_66559fa571be4748b005f229c7089c09 [id=5431f5161ddf40d083b94b5fcb6b7561]) successfully deleted
I20250112 02:52:35.881500 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 20) took 1798 ms. Trace:
I20250112 02:52:35.882167 2036 rpcz_store.cc:276] 0112 02:52:34.082725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.082812 (+ 87us) service_pool.cc:225] Handling call
0112 02:52:35.881377 (+1798565us) ts_tablet_manager.cc:1145] Deleting tablet f776e2a6ec704cf9b64483bc7d5b1dff
0112 02:52:35.881386 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:35.881483 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1604621}
W20250112 02:52:35.882930 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet f776e2a6ec704cf9b64483bc7d5b1dff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f776e2a6ec704cf9b64483bc7d5b1dff already in progress: creating tablet
I20250112 02:52:35.883571 2036 tablet_replica.cc:331] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:35.891878 1016 tablet_service.cc:1467] Processing CreateTablet for tablet f559dedfb7854cf890bef362c50b4213 (DEFAULT_TABLE table=default.Table_7921b99e24f84cceb58128f18925ad49 [id=0525f87eccbb4a1d8726d18a00f9a4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.892022 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 37d38f5e525440f8a0f7822620933435 (DEFAULT_TABLE table=default.Table_39f67a35c5c640dd8cbbcea89df5e4fc [id=9d267774bf59428e854651956fbd79be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.893841 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 2796289c7e8f4b45b26508c50b57c640 (DEFAULT_TABLE table=default.Table_23c987a24fed4ceeb90fd25f0367b08f [id=a9f367de65284002a6a8ea94fc8cb7d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.894527 1177 tablet_bootstrap.cc:654] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:35.896306 2216 tablet.cc:1721] T 2c03b5e612d64314b210c11686fdfa2a P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250112 02:52:35.897775 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 430a8652c53e4801a1c72de529c2adcb (DEFAULT_TABLE table=default.Table_b645d30cd43d4f09a3ac34db205a80f9 [id=aa437a13d9bc4b05a2f99a6e494ef36a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.901872 2036 raft_consensus.cc:2238] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:35.902017 2036 pending_rounds.cc:62] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d: Trying to abort 1 pending ops.
I20250112 02:52:35.902070 2036 pending_rounds.cc:69] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319848487952384 op_type: NO_OP noop_request { }
I20250112 02:52:35.902314 2036 raft_consensus.cc:2883] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:35.902419 2036 raft_consensus.cc:2267] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:35.909369 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b351c563d749df806ab5d0331a4ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:35.909794 2036 ts_tablet_manager.cc:1905] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:35.913865 1177 tablet_bootstrap.cc:492] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:35.914098 1177 ts_tablet_manager.cc:1397] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20250112 02:52:35.915359 1177 raft_consensus.cc:357] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.915719 1177 raft_consensus.cc:383] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:35.915800 1177 raft_consensus.cc:738] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:35.916144 1177 consensus_queue.cc:260] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.917721 1177 ts_tablet_manager.cc:1428] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 02:52:35.918736 1177 tablet_bootstrap.cc:492] T f7cd0927c6a4401cbe69f55d4db9963a P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:35.920390 2222 raft_consensus.cc:491] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:35.920511 2222 raft_consensus.cc:513] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.921090 2222 leader_election.cc:290] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:35.921159 2003 ts_tablet_manager.cc:1918] T bb217afc08304c0ea35dee3120347ba1 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:35.921240 2003 log.cc:1198] T bb217afc08304c0ea35dee3120347ba1 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/bb217afc08304c0ea35dee3120347ba1
I20250112 02:52:35.921602 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b61f16f3d601436bb4b32344de28c1a7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:35.922178 855 leader_election.cc:343] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:35.924605 714 leader_election.cc:304] T 89b70dce4d1d4e29997a18297ac0f82a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:35.924826 2179 raft_consensus.cc:3054] T 89b70dce4d1d4e29997a18297ac0f82a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:35.924971 714 leader_election.cc:304] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:35.927083 2179 raft_consensus.cc:2743] T 89b70dce4d1d4e29997a18297ac0f82a P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:35.927229 2179 raft_consensus.cc:2743] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:35.929685 1778 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:52:35.929970 1028 tablet_service.cc:1467] Processing CreateTablet for tablet acafc4f579534a69a8f307ca4cf72e12 (DEFAULT_TABLE table=default.Table_cc762fcab6cf493ab58b3d1defecf106 [id=1b44f2cf98e246b0b3afbd1e3fa9676c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.931778 1778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:52:35.931985 2003 ts_tablet_manager.cc:1939] T bb217afc08304c0ea35dee3120347ba1 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:35.932317 1025 tablet_service.cc:1467] Processing CreateTablet for tablet a4328137407843e1879493ef0c56ffc8 (DEFAULT_TABLE table=default.Table_c3e55b9696194ba58cbc9f1bb391b27d [id=9049d21b5e854418b070560cecda2eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.932518 1778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:52:35.932765 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 31) took 1845 ms. Trace:
I20250112 02:52:35.932832 2003 rpcz_store.cc:276] 0112 02:52:34.087163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.087854 (+ 691us) service_pool.cc:225] Handling call
0112 02:52:35.872733 (+1784879us) ts_tablet_manager.cc:1145] Deleting tablet bb217afc08304c0ea35dee3120347ba1
0112 02:52:35.872741 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:35.890460 (+ 17719us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.921073 (+ 30613us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.932746 (+ 11673us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1775439}
I20250112 02:52:35.929947 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 003a3b4705e943c6a900762ec11ddd47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:35.933248 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet bb217afc08304c0ea35dee3120347ba1 (table default.Table_94d3a2be9a344af082b69ecfa503d4c0 [id=da2e1bf15b21475085a3da6b16aecc4c]) successfully deleted
I20250112 02:52:35.933395 1177 tablet_bootstrap.cc:654] T f7cd0927c6a4401cbe69f55d4db9963a P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:35.934221 2003 tablet_replica.cc:331] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:35.934405 2003 raft_consensus.cc:2238] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:35.934649 2003 raft_consensus.cc:2267] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:35.935261 2003 ts_tablet_manager.cc:1905] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:35.953150 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 940affdc2c484b7d9fd2de17f9d1cea7 (DEFAULT_TABLE table=default.Table_9afc6941825a45bc8f98bf373ff5f27c [id=c76eb62dc8bf4b97b675787374189057]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.954061 1021 tablet_service.cc:1467] Processing CreateTablet for tablet cd52261c4a414e7d93a528e2eddfa9db (DEFAULT_TABLE table=default.Table_70d1488de98d4380b8024db852f58991 [id=407e1e8894524b7682f498b3904b06e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:35.954116 2179 raft_consensus.cc:491] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:35.954218 2179 raft_consensus.cc:513] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.954816 2179 leader_election.cc:290] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:35.955255 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2730634a5cfd4c2fb5c4e30100c556d4" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:35.955497 910 raft_consensus.cc:2463] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:35.955893 713 leader_election.cc:304] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:35.956152 2179 raft_consensus.cc:2798] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:35.956281 2179 raft_consensus.cc:491] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:35.956404 2179 raft_consensus.cc:3054] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:35.956380 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003a3b4705e943c6a900762ec11ddd47" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:35.957911 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b61f16f3d601436bb4b32344de28c1a7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:35.958743 2179 raft_consensus.cc:513] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.960489 2179 leader_election.cc:290] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:35.960819 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2730634a5cfd4c2fb5c4e30100c556d4" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:35.961019 910 raft_consensus.cc:3054] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:35.963949 910 raft_consensus.cc:2463] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:35.964349 713 leader_election.cc:304] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:35.964666 2179 raft_consensus.cc:2798] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:35.965096 2179 raft_consensus.cc:695] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:35.965848 2179 consensus_queue.cc:237] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.966624 2003 ts_tablet_manager.cc:1918] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:35.966743 2003 log.cc:1198] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/161fe26b9c0e4543b7aeb0021506d72a
I20250112 02:52:35.967185 2003 ts_tablet_manager.cc:1939] T 161fe26b9c0e4543b7aeb0021506d72a P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:35.968047 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 26) took 1894 ms. Trace:
I20250112 02:52:35.968152 2003 rpcz_store.cc:276] 0112 02:52:34.073514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.073556 (+ 42us) service_pool.cc:225] Handling call
0112 02:52:35.934180 (+1860624us) ts_tablet_manager.cc:1145] Deleting tablet 161fe26b9c0e4543b7aeb0021506d72a
0112 02:52:35.934189 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:35.946732 (+ 12543us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.966519 (+ 19787us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.968026 (+ 1507us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1836817}
I20250112 02:52:35.968580 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 161fe26b9c0e4543b7aeb0021506d72a (table default.Table_981a70d4d3184fd8a99e6b81ae1caab7 [id=8016eb490f574ad8a5d3ee598fecaeb5]) successfully deleted
I20250112 02:52:35.974632 2003 tablet_replica.cc:331] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:35.974843 2003 raft_consensus.cc:2238] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:35.975082 2003 raft_consensus.cc:2267] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:35.975661 2003 ts_tablet_manager.cc:1905] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:35.979970 854 leader_election.cc:343] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:35.980077 854 leader_election.cc:304] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:35.980628 854 leader_election.cc:343] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:35.980734 854 leader_election.cc:304] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:35.980994 2117 raft_consensus.cc:2743] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:35.981163 2117 raft_consensus.cc:2743] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:35.981446 854 leader_election.cc:343] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:35.981528 854 leader_election.cc:304] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:35.981567 2036 ts_tablet_manager.cc:1918] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:35.981856 2036 log.cc:1198] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/4b599e983ec04f53a440d1ab828d12ab
W20250112 02:52:35.982175 854 leader_election.cc:343] T 763f77385e404c149c16edabe469c299 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:35.982213 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6117dc8b32a64535a2f4167a03bed0d2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:35.982225 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet cb724feccc7f4fbc846fb6fa61be3385 (table default.Table_43048dfe7bf24df5a7254bdb6854101e [id=81a09e598ee34508aa88abfc6b44a2bd]) successfully deleted
I20250112 02:52:35.982260 854 leader_election.cc:304] T 763f77385e404c149c16edabe469c299 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:35.982337 910 raft_consensus.cc:3054] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:35.982451 2223 raft_consensus.cc:2743] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:35.982566 854 leader_election.cc:343] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:35.982580 2223 raft_consensus.cc:2743] T 763f77385e404c149c16edabe469c299 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:35.982653 854 leader_election.cc:304] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:35.982795 2223 raft_consensus.cc:2743] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:35.982856 2036 ts_tablet_manager.cc:1939] T 4b599e983ec04f53a440d1ab828d12ab P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:35.984613 910 raft_consensus.cc:2463] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:35.984889 2047 raft_consensus.cc:491] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:35.985000 2047 raft_consensus.cc:513] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:35.985561 2047 leader_election.cc:290] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:35.986132 713 leader_election.cc:343] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 1dda52f012794493a102ed1abc718f5b
W20250112 02:52:35.986101 854 leader_election.cc:343] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:35.986248 854 leader_election.cc:304] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:35.986613 2117 raft_consensus.cc:2743] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:35.987735 854 leader_election.cc:343] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:35.988024 854 leader_election.cc:304] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:35.988444 2117 raft_consensus.cc:2743] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:35.995731 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 45) took 1790 ms. Trace:
I20250112 02:52:35.997886 2003 ts_tablet_manager.cc:1918] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:35.998431 2003 log.cc:1198] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/41e166b3f8414ababbd4a3f07fb90c4c
I20250112 02:52:35.998914 2003 ts_tablet_manager.cc:1939] T 41e166b3f8414ababbd4a3f07fb90c4c P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:35.997131 2036 rpcz_store.cc:276] 0112 02:52:34.204907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.276661 (+ 71754us) service_pool.cc:225] Handling call
0112 02:52:35.883530 (+1606869us) ts_tablet_manager.cc:1145] Deleting tablet 4b599e983ec04f53a440d1ab828d12ab
0112 02:52:35.883540 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:35.912777 (+ 29237us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.981122 (+ 68345us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.995713 (+ 14591us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1606755}
I20250112 02:52:36.000409 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 36) took 1911 ms. Trace:
I20250112 02:52:36.000427 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 4b599e983ec04f53a440d1ab828d12ab (table default.Table_feacb3254bbe4765a4e0a4908fbc694c [id=f396d2dc8dbd47729f49f7511467d481]) successfully deleted
I20250112 02:52:36.003069 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 46) took 1797 ms. Trace:
I20250112 02:52:36.004108 2003 rpcz_store.cc:276] 0112 02:52:34.088639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.088723 (+ 84us) service_pool.cc:225] Handling call
0112 02:52:35.974579 (+1885856us) ts_tablet_manager.cc:1145] Deleting tablet 41e166b3f8414ababbd4a3f07fb90c4c
0112 02:52:35.974592 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:35.992423 (+ 17831us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:35.997817 (+ 5394us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.000386 (+ 2569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1866674}
I20250112 02:52:36.004809 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 41e166b3f8414ababbd4a3f07fb90c4c (table default.Table_11b4065cfb0449b68950d63e65f566a2 [id=0e325561bbdb468e8a64ca9808e2f3cf]) successfully deleted
I20250112 02:52:36.004881 2003 tablet_replica.cc:331] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:36.005079 2003 raft_consensus.cc:2238] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.005666 2003 raft_consensus.cc:2267] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.006363 2003 ts_tablet_manager.cc:1905] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.004287 2036 rpcz_store.cc:276] 0112 02:52:34.205466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.276759 (+ 71293us) service_pool.cc:225] Handling call
0112 02:52:36.001605 (+1724846us) ts_tablet_manager.cc:1145] Deleting tablet b6d84027ad6d40a5be5afcdd854431a5
0112 02:52:36.001886 (+ 281us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.003054 (+ 1168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1724232}
W20250112 02:52:36.008867 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet b6d84027ad6d40a5be5afcdd854431a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6d84027ad6d40a5be5afcdd854431a5 already in progress: creating tablet
I20250112 02:52:36.011284 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 47) took 1805 ms. Trace:
I20250112 02:52:36.016618 2003 ts_tablet_manager.cc:1918] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.012982 2036 rpcz_store.cc:276] 0112 02:52:34.205849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.276778 (+ 70929us) service_pool.cc:225] Handling call
0112 02:52:36.009810 (+1733032us) ts_tablet_manager.cc:1145] Deleting tablet b2ccb88108f24e36ba4b7ebf36398950
0112 02:52:36.010092 (+ 282us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.010979 (+ 887us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1732639}
W20250112 02:52:36.021092 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet b2ccb88108f24e36ba4b7ebf36398950 because the tablet was not found. No further retry: Not found: Tablet not found: b2ccb88108f24e36ba4b7ebf36398950
I20250112 02:52:36.023257 2036 tablet_replica.cc:331] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:36.024951 2003 log.cc:1198] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/4c0c2fad4f3243238641aae56b305bb5
I20250112 02:52:36.025468 2003 ts_tablet_manager.cc:1939] T 4c0c2fad4f3243238641aae56b305bb5 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:36.027052 2036 raft_consensus.cc:2238] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.028909 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 33) took 1941 ms. Trace:
I20250112 02:52:36.029824 2003 rpcz_store.cc:276] 0112 02:52:34.087379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.087734 (+ 355us) service_pool.cc:225] Handling call
0112 02:52:36.004835 (+1917101us) ts_tablet_manager.cc:1145] Deleting tablet 4c0c2fad4f3243238641aae56b305bb5
0112 02:52:36.004846 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.013561 (+ 8715us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.016551 (+ 2990us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.028890 (+ 12339us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1891881}
I20250112 02:52:36.030568 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 4c0c2fad4f3243238641aae56b305bb5 (table default.Table_82662e02aee0459dad1b621df36f58f0 [id=50a3f0607e95430594c6769983f4e31b]) successfully deleted
I20250112 02:52:36.030663 2003 tablet_replica.cc:331] T 7f4e4f1e451b48dc9cdd780300f6888c P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:36.030864 2003 raft_consensus.cc:2238] T 7f4e4f1e451b48dc9cdd780300f6888c P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.031484 2003 raft_consensus.cc:2267] T 7f4e4f1e451b48dc9cdd780300f6888c P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.029992 2036 raft_consensus.cc:2267] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.032276 2003 ts_tablet_manager.cc:1905] T 7f4e4f1e451b48dc9cdd780300f6888c P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.036151 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2730634a5cfd4c2fb5c4e30100c556d4" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:36.036602 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2730634a5cfd4c2fb5c4e30100c556d4" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:36.039685 2036 ts_tablet_manager.cc:1905] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.046906 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dda52f012794493a102ed1abc718f5b" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:36.047099 1053 raft_consensus.cc:2463] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:36.047967 2219 tablet.cc:1721] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:52:36.056593 439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250112 02:52:36.063216 2185 tablet.cc:1721] T 3f5b418c17084420b867498fda79fb0a P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:36.075253 2117 raft_consensus.cc:491] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.075407 2117 raft_consensus.cc:513] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:36.076086 2117 leader_election.cc:290] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:36.076427 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc414d6420c41839621ab858067548b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:36.076967 855 leader_election.cc:343] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:36.079496 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet bb4ef0220dde43569452379639759db5 consensus state has no leader
I20250112 02:52:36.095813 2228 consensus_queue.cc:1035] T 88d69632845547f588662dd56c8c194e P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.259s
I20250112 02:52:36.096787 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c1ae3f34b434ee5bc3519cce8d283a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.097023 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2796289c7e8f4b45b26508c50b57c640. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.103621 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5524d75599f04becbe6a94426adb7026. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.105211 2219 tablet.cc:1721] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:36.107589 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f559dedfb7854cf890bef362c50b4213. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.108951 2036 ts_tablet_manager.cc:1918] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:36.109047 2036 log.cc:1198] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/bb217afc08304c0ea35dee3120347ba1
I20250112 02:52:36.109815 2036 ts_tablet_manager.cc:1939] T bb217afc08304c0ea35dee3120347ba1 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:36.110486 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 48) took 1904 ms. Trace:
I20250112 02:52:36.111631 2036 rpcz_store.cc:276] 0112 02:52:34.206499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.276868 (+ 70369us) service_pool.cc:225] Handling call
0112 02:52:36.022414 (+1745546us) ts_tablet_manager.cc:1145] Deleting tablet bb217afc08304c0ea35dee3120347ba1
0112 02:52:36.022426 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.068675 (+ 46249us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.096201 (+ 27526us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.110467 (+ 14266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1745032}
I20250112 02:52:36.111930 2036 tablet_replica.cc:331] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:36.112124 2036 raft_consensus.cc:2238] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.112288 2036 raft_consensus.cc:2267] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.112766 2036 ts_tablet_manager.cc:1905] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.112854 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet bb217afc08304c0ea35dee3120347ba1 (table default.Table_94d3a2be9a344af082b69ecfa503d4c0 [id=da2e1bf15b21475085a3da6b16aecc4c]) successfully deleted
I20250112 02:52:36.115617 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4328137407843e1879493ef0c56ffc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.121671 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acafc4f579534a69a8f307ca4cf72e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.124953 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 940affdc2c484b7d9fd2de17f9d1cea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.130463 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd52261c4a414e7d93a528e2eddfa9db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.134232 2036 ts_tablet_manager.cc:1918] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.134336 2036 log.cc:1198] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/ea33e5c1a1fb46599c5836a36b130051
I20250112 02:52:36.134500 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 430a8652c53e4801a1c72de529c2adcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.134697 2036 ts_tablet_manager.cc:1939] T ea33e5c1a1fb46599c5836a36b130051 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:36.136669 1012 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50840 (request call id 105) took 1080 ms. Trace:
I20250112 02:52:36.142459 1181 tablet_bootstrap.cc:654] T 781fe3514f4c4f80b1ee49278502ccc1 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.143350 1012 rpcz_store.cc:276] 0112 02:52:35.056681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:35.058255 (+ 1574us) service_pool.cc:225] Handling call
0112 02:52:35.217420 (+159165us) spinlock_profiling.cc:243] Waited 77.6 ms on lock 0x61800016008c. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8ccd08ee 00007f2e8cacf1df 00007f2e8f4dd948 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78
0112 02:52:35.298218 (+ 80798us) spinlock_profiling.cc:243] Waited 44.2 ms on lock 0x6160001e1054. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.349776 (+ 51558us) spinlock_profiling.cc:243] Waited 1.37 ms on lock 0x6160000e1830. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8cacf068 00007f2e8f4dd948 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.444953 (+ 95177us) spinlock_profiling.cc:243] Waited 73.7 ms on lock 0x616000142730. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8cacf068 00007f2e8f4dd948 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.704426 (+259473us) spinlock_profiling.cc:243] Waited 89.2 ms on lock 0x616000126454. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.934269 (+229843us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x616000113854. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:36.136624 (+202355us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1165808512}
I20250112 02:52:36.139580 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37d38f5e525440f8a0f7822620933435. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.141695 1013 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50840 (request call id 103) took 1380 ms. Trace:
I20250112 02:52:36.144359 1013 rpcz_store.cc:276] 0112 02:52:34.761172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.761415 (+ 243us) service_pool.cc:225] Handling call
0112 02:52:35.126313 (+364898us) spinlock_profiling.cc:243] Waited 353 ms on lock 0x6160000c3d54. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.298170 (+171857us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x6160001e1054. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.455586 (+157416us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x616000142730. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8cacf068 00007f2e8f4dd948 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.934677 (+479091us) spinlock_profiling.cc:243] Waited 475 ms on lock 0x616000113854. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:36.141648 (+206971us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2462822144}
I20250112 02:52:36.143241 1011 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50840 (request call id 102) took 1433 ms. Trace:
I20250112 02:52:36.144791 1011 rpcz_store.cc:276] 0112 02:52:34.710153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.710355 (+ 202us) service_pool.cc:225] Handling call
0112 02:52:34.759754 (+ 49399us) spinlock_profiling.cc:243] Waited 4.12 ms on lock 0x6160000c5554. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.125606 (+365852us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x6160000c3d54. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.299376 (+173770us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x6160001e1054. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.445133 (+145757us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x616000142730. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8cacf068 00007f2e8f4dd948 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.705035 (+259902us) spinlock_profiling.cc:243] Waited 48.3 ms on lock 0x616000126454. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.932799 (+227764us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x616000113854. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:36.143219 (+210420us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1958168704}
W20250112 02:52:36.144948 714 leader_election.cc:343] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:36.145275 714 leader_election.cc:343] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:36.145637 714 leader_election.cc:304] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:36.145838 2190 raft_consensus.cc:2798] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:36.145907 2190 raft_consensus.cc:491] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:36.145954 2190 raft_consensus.cc:3054] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:36.146648 991 leader_election.cc:343] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: 1726d61cc36a4deb83c5d604e8daf4f0
I20250112 02:52:36.146762 991 leader_election.cc:304] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:36.147035 989 leader_election.cc:304] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:36.148447 2190 raft_consensus.cc:513] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.149056 2190 leader_election.cc:290] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:36.135409 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 49) took 1928 ms. Trace:
W20250112 02:52:36.150413 713 leader_election.cc:343] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 1dda52f012794493a102ed1abc718f5b
I20250112 02:52:36.150606 2228 raft_consensus.cc:2743] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:36.150568 2036 rpcz_store.cc:276] 0112 02:52:34.207269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.276916 (+ 69647us) service_pool.cc:225] Handling call
0112 02:52:36.111885 (+1834969us) ts_tablet_manager.cc:1145] Deleting tablet ea33e5c1a1fb46599c5836a36b130051
0112 02:52:36.111895 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.115301 (+ 3406us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.134160 (+ 18859us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.135392 (+ 1232us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1834861}
W20250112 02:52:36.150852 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 313978us
I20250112 02:52:36.151206 2036 tablet_replica.cc:331] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:36.151353 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet ea33e5c1a1fb46599c5836a36b130051 (table default.Table_7abcf14838c744109a08985ba6ce3e0d [id=5f25c42be8924b41be7f2f268d068e04]) successfully deleted
I20250112 02:52:36.151736 2228 raft_consensus.cc:2798] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:36.151804 2228 raft_consensus.cc:491] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:36.151881 2228 raft_consensus.cc:3054] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.151708 2036 raft_consensus.cc:2238] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:36.152467 2036 raft_consensus.cc:2267] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.154325 2228 raft_consensus.cc:513] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.155334 2228 leader_election.cc:290] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:36.165959 2036 ts_tablet_manager.cc:1905] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:36.151070 991 leader_election.cc:343] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:36.167186 991 leader_election.cc:304] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:36.167773 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbe30f9860e2492b90e8cedf8ff9daba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.171149 2036 ts_tablet_manager.cc:1918] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.171147 1015 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50840 (request call id 104) took 1401 ms. Trace:
I20250112 02:52:36.171252 2036 log.cc:1198] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/6913b0ac9ceb4dbc9cdf591f159134f7
I20250112 02:52:36.171242 1015 rpcz_store.cc:276] 0112 02:52:34.770030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.770077 (+ 47us) service_pool.cc:225] Handling call
0112 02:52:35.126090 (+356013us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x6160000c3d54. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.151014 (+ 24924us) spinlock_profiling.cc:243] Waited 3.77 ms on lock 0x61800016008c. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8ccd08ee 00007f2e8cacf1df 00007f2e8f4dd948 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78
0112 02:52:35.299430 (+148416us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x6160001e1054. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.444993 (+145563us) spinlock_profiling.cc:243] Waited 60.2 ms on lock 0x616000142730. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8cacf068 00007f2e8f4dd948 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.704708 (+259715us) spinlock_profiling.cc:243] Waited 62.3 ms on lock 0x616000126454. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.934511 (+229803us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x616000113854. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:36.171123 (+236612us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1828329600}
I20250112 02:52:36.174670 2003 ts_tablet_manager.cc:1918] T 7f4e4f1e451b48dc9cdd780300f6888c P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.176313 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fe57d374eb44c3fb1532484e214b654. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.178550 2003 log.cc:1198] T 7f4e4f1e451b48dc9cdd780300f6888c P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/7f4e4f1e451b48dc9cdd780300f6888c
I20250112 02:52:36.179203 2003 ts_tablet_manager.cc:1939] T 7f4e4f1e451b48dc9cdd780300f6888c P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:36.182323 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1123077650c4e89bffddd6d1264e428. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:36.183851 855 consensus_peers.cc:487] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:36.184747 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 40) took 2067 ms. Trace:
I20250112 02:52:36.184882 2003 rpcz_store.cc:276] 0112 02:52:34.117554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.117672 (+ 118us) service_pool.cc:225] Handling call
0112 02:52:36.030615 (+1912943us) ts_tablet_manager.cc:1145] Deleting tablet 7f4e4f1e451b48dc9cdd780300f6888c
0112 02:52:36.030625 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.047351 (+ 16726us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.174579 (+127228us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.184729 (+ 10150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1910690}
I20250112 02:52:36.185642 2003 tablet_replica.cc:331] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:36.185613 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 7f4e4f1e451b48dc9cdd780300f6888c (table default.Table_096703f878244b8dac2dcf187fd23fd6 [id=e1779f88b20942eaa2c640ac24946fff]) successfully deleted
I20250112 02:52:36.186697 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99407ead9cf547e3bbe78c0d0b327e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.191496 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f52b838214c54ac69ad30dac6e56669a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.194763 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b741e4f6abd4695b3ba1d7a667b6046. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.195022 2003 raft_consensus.cc:2238] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.195209 2003 raft_consensus.cc:2267] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.198726 2036 ts_tablet_manager.cc:1939] T 6913b0ac9ceb4dbc9cdf591f159134f7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:36.195670 1177 tablet_bootstrap.cc:492] T f7cd0927c6a4401cbe69f55d4db9963a P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:36.198948 1177 ts_tablet_manager.cc:1397] T f7cd0927c6a4401cbe69f55d4db9963a P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.281s user 0.028s sys 0.073s
I20250112 02:52:36.199344 2003 ts_tablet_manager.cc:1905] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.199410 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56ddc025a8314f6f84d229191eb56692. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.199380 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 50) took 1991 ms. Trace:
I20250112 02:52:36.199451 2036 rpcz_store.cc:276] 0112 02:52:34.207819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.277011 (+ 69192us) service_pool.cc:225] Handling call
0112 02:52:36.151162 (+1874151us) ts_tablet_manager.cc:1145] Deleting tablet 6913b0ac9ceb4dbc9cdf591f159134f7
0112 02:52:36.151173 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.168888 (+ 17715us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.171080 (+ 2192us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.199351 (+ 28271us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1874026}
I20250112 02:52:36.199625 1177 raft_consensus.cc:357] T f7cd0927c6a4401cbe69f55d4db9963a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.199767 2036 tablet_replica.cc:331] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:36.199800 1177 raft_consensus.cc:383] T f7cd0927c6a4401cbe69f55d4db9963a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.199851 1177 raft_consensus.cc:738] T f7cd0927c6a4401cbe69f55d4db9963a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.199908 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 6913b0ac9ceb4dbc9cdf591f159134f7 (table default.Table_f4d612cd6a524bc493147493c5a4a284 [id=4ac16bc62bbe4ed7a977d22c92ad580d]) successfully deleted
I20250112 02:52:36.199940 2036 raft_consensus.cc:2238] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.200064 2036 raft_consensus.cc:2267] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.200018 1177 consensus_queue.cc:260] T f7cd0927c6a4401cbe69f55d4db9963a P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.200574 1177 ts_tablet_manager.cc:1428] T f7cd0927c6a4401cbe69f55d4db9963a P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:36.200790 1177 tablet_bootstrap.cc:492] T 8ee98f055024429eafaa29bb961cab87 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:36.202066 1177 tablet_bootstrap.cc:654] T 8ee98f055024429eafaa29bb961cab87 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.202283 2036 ts_tablet_manager.cc:1905] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.204825 1014 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50840 (request call id 101) took 1536 ms. Trace:
I20250112 02:52:36.204958 1014 rpcz_store.cc:276] 0112 02:52:34.668840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.669266 (+ 426us) service_pool.cc:225] Handling call
0112 02:52:34.758177 (+ 88911us) spinlock_profiling.cc:243] Waited 4.81 ms on lock 0x6160000c5554. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.125529 (+367352us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x6160000c3d54. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.298667 (+173138us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x6160001e1054. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.348822 (+ 50155us) spinlock_profiling.cc:243] Waited 974 us on lock 0x6160000e1830. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8cacf068 00007f2e8f4dd948 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.444792 (+ 95970us) spinlock_profiling.cc:243] Waited 75.7 ms on lock 0x616000142730. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8cacf068 00007f2e8f4dd948 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.643108 (+198316us) spinlock_profiling.cc:243] Waited 3.76 ms on lock 0x60c000215e78. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b19d89b 00007f2e8b15a783 00007f2e8cacfe28 00007f2e8cacf40e 00007f2e8f4dd948 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425
0112 02:52:35.701644 (+ 58536us) spinlock_profiling.cc:243] Waited 43.4 ms on lock 0x616000126454. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:35.935508 (+233864us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x616000113854. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:36.204805 (+269297us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1888298112}
I20250112 02:52:36.205492 2228 consensus_queue.cc:1035] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:52:36.205989 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 47757.7us
I20250112 02:52:36.209496 2228 consensus_queue.cc:1035] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:36.211328 2024 pending_rounds.cc:62] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6: Trying to abort 1 pending ops.
I20250112 02:52:36.211458 2024 pending_rounds.cc:69] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319852493385728 op_type: NO_OP noop_request { }
I20250112 02:52:36.211624 2024 raft_consensus.cc:2883] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:36.211787 2024 raft_consensus.cc:2267] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.212463 2024 ts_tablet_manager.cc:1905] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:36.214545 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 63469.6us
I20250112 02:52:36.214710 2217 raft_consensus.cc:2743] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:36.216157 2003 ts_tablet_manager.cc:1918] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.216289 2003 log.cc:1198] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/42cb7e7a41634cf1bdb978ac9d2befcd
I20250112 02:52:36.216671 2003 ts_tablet_manager.cc:1939] T 42cb7e7a41634cf1bdb978ac9d2befcd P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
W20250112 02:52:36.217489 991 consensus_peers.cc:487] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: daef3a58605e40978f7389698b22265d. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:36.217656 1177 tablet_bootstrap.cc:492] T 8ee98f055024429eafaa29bb961cab87 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:36.217772 1177 ts_tablet_manager.cc:1397] T 8ee98f055024429eafaa29bb961cab87 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.005s
I20250112 02:52:36.218335 1177 raft_consensus.cc:357] T 8ee98f055024429eafaa29bb961cab87 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.218495 1177 raft_consensus.cc:383] T 8ee98f055024429eafaa29bb961cab87 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.218531 1177 raft_consensus.cc:738] T 8ee98f055024429eafaa29bb961cab87 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.217780 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 42) took 2097 ms. Trace:
I20250112 02:52:36.218681 1177 consensus_queue.cc:260] T 8ee98f055024429eafaa29bb961cab87 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.219316 1177 ts_tablet_manager.cc:1428] T 8ee98f055024429eafaa29bb961cab87 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 02:52:36.219563 1177 tablet_bootstrap.cc:492] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:36.218691 2003 rpcz_store.cc:276] 0112 02:52:34.119862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.120027 (+ 165us) service_pool.cc:225] Handling call
0112 02:52:36.185594 (+2065567us) ts_tablet_manager.cc:1145] Deleting tablet 42cb7e7a41634cf1bdb978ac9d2befcd
0112 02:52:36.185605 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.204961 (+ 19356us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.216055 (+ 11094us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.217766 (+ 1711us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2063663}
I20250112 02:52:36.220026 2003 tablet_replica.cc:331] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:36.220187 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 42cb7e7a41634cf1bdb978ac9d2befcd (table default.Table_3530ac60f1e04607aafce2de389968df [id=8245f0e407da4a88ad69764267feed0e]) successfully deleted
I20250112 02:52:36.220345 2003 raft_consensus.cc:2238] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.220535 2003 raft_consensus.cc:2267] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.220970 2003 ts_tablet_manager.cc:1905] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.221184 1177 tablet_bootstrap.cc:654] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.226697 1177 tablet_bootstrap.cc:492] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:36.226805 1177 ts_tablet_manager.cc:1397] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20250112 02:52:36.227494 1177 raft_consensus.cc:357] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.227677 1177 raft_consensus.cc:383] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.227749 1177 raft_consensus.cc:738] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.227948 1177 consensus_queue.cc:260] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.230443 1177 ts_tablet_manager.cc:1428] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:36.230647 1177 tablet_bootstrap.cc:492] T 1811ce8982fd4003a35e89ea34e32746 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:36.230890 2003 ts_tablet_manager.cc:1918] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.230978 2003 log.cc:1198] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/35f59617b34046568dd83bbb7cdc42d2
I20250112 02:52:36.231418 2003 ts_tablet_manager.cc:1939] T 35f59617b34046568dd83bbb7cdc42d2 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:36.232110 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 43) took 2112 ms. Trace:
I20250112 02:52:36.232271 1177 tablet_bootstrap.cc:654] T 1811ce8982fd4003a35e89ea34e32746 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:36.234853 991 leader_election.cc:343] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:36.237071 1177 tablet_bootstrap.cc:492] T 1811ce8982fd4003a35e89ea34e32746 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:36.237181 1177 ts_tablet_manager.cc:1397] T 1811ce8982fd4003a35e89ea34e32746 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20250112 02:52:36.237756 1177 raft_consensus.cc:357] T 1811ce8982fd4003a35e89ea34e32746 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.237905 1177 raft_consensus.cc:383] T 1811ce8982fd4003a35e89ea34e32746 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.237949 1177 raft_consensus.cc:738] T 1811ce8982fd4003a35e89ea34e32746 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.232213 2003 rpcz_store.cc:276] 0112 02:52:34.119973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.120013 (+ 40us) service_pool.cc:225] Handling call
0112 02:52:36.219989 (+2099976us) ts_tablet_manager.cc:1145] Deleting tablet 35f59617b34046568dd83bbb7cdc42d2
0112 02:52:36.219998 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.227910 (+ 7912us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.230834 (+ 2924us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.232093 (+ 1259us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2096867}
I20250112 02:52:36.239733 2003 tablet_replica.cc:331] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:36.239768 1177 consensus_queue.cc:260] T 1811ce8982fd4003a35e89ea34e32746 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.239869 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 35f59617b34046568dd83bbb7cdc42d2 (table default.Table_cfef2f0eb8dd4927b6d0af0a8c3c739c [id=a0145eb8ddc14728b448e5c08a4d9311]) successfully deleted
I20250112 02:52:36.239903 2003 raft_consensus.cc:2238] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:36.240543 1177 ts_tablet_manager.cc:1428] T 1811ce8982fd4003a35e89ea34e32746 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 02:52:36.241039 1177 tablet_bootstrap.cc:492] T 5cc414d6420c41839621ab858067548b P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:36.242900 1177 tablet_bootstrap.cc:654] T 5cc414d6420c41839621ab858067548b P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.245492 2003 raft_consensus.cc:2267] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.246276 2003 ts_tablet_manager.cc:1905] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.248163 1177 tablet_bootstrap.cc:492] T 5cc414d6420c41839621ab858067548b P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:36.248267 1177 ts_tablet_manager.cc:1397] T 5cc414d6420c41839621ab858067548b P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20250112 02:52:36.248847 1177 raft_consensus.cc:357] T 5cc414d6420c41839621ab858067548b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:36.250447 1177 raft_consensus.cc:383] T 5cc414d6420c41839621ab858067548b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.250515 1177 raft_consensus.cc:738] T 5cc414d6420c41839621ab858067548b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.250687 1177 consensus_queue.cc:260] T 5cc414d6420c41839621ab858067548b P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:36.252211 1177 ts_tablet_manager.cc:1428] T 5cc414d6420c41839621ab858067548b P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250112 02:52:36.252430 1177 tablet_bootstrap.cc:492] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:36.224803 2024 ts_tablet_manager.cc:1918] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.254954 1177 tablet_bootstrap.cc:654] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.255096 2024 log.cc:1198] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/1a1e3af2883341e9b957f6ed748e604d
I20250112 02:52:36.255530 2024 ts_tablet_manager.cc:1939] T 1a1e3af2883341e9b957f6ed748e604d P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:36.260749 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 37) took 1844 ms. Trace:
I20250112 02:52:36.260805 1177 tablet_bootstrap.cc:492] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:36.260895 1177 ts_tablet_manager.cc:1397] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250112 02:52:36.262046 1177 raft_consensus.cc:357] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.262239 1177 raft_consensus.cc:383] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.262290 1177 raft_consensus.cc:738] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.228333 1173 tablet_bootstrap.cc:492] T 5cc414d6420c41839621ab858067548b P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:36.261931 2207 consensus_queue.cc:1035] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:36.262454 1177 consensus_queue.cc:260] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.262547 1173 ts_tablet_manager.cc:1397] T 5cc414d6420c41839621ab858067548b P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.592s user 0.010s sys 0.003s
I20250112 02:52:36.262781 989 leader_election.cc:304] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:36.263316 1173 raft_consensus.cc:357] T 5cc414d6420c41839621ab858067548b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:36.263485 1173 raft_consensus.cc:383] T 5cc414d6420c41839621ab858067548b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.263545 1173 raft_consensus.cc:738] T 5cc414d6420c41839621ab858067548b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.263720 1173 consensus_queue.cc:260] T 5cc414d6420c41839621ab858067548b P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:36.264184 1173 ts_tablet_manager.cc:1428] T 5cc414d6420c41839621ab858067548b P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:36.264425 1173 tablet_bootstrap.cc:492] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:36.265164 1177 ts_tablet_manager.cc:1428] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250112 02:52:36.265393 1177 tablet_bootstrap.cc:492] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:36.265543 1181 tablet_bootstrap.cc:492] T 781fe3514f4c4f80b1ee49278502ccc1 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:36.265709 1181 ts_tablet_manager.cc:1397] T 781fe3514f4c4f80b1ee49278502ccc1 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.443s user 0.007s sys 0.000s
I20250112 02:52:36.266471 1181 raft_consensus.cc:357] T 781fe3514f4c4f80b1ee49278502ccc1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.267387 1173 tablet_bootstrap.cc:654] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.267779 2003 ts_tablet_manager.cc:1918] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.267884 2003 log.cc:1198] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/1a8a178bafcb457f9b0e9587df560493
I20250112 02:52:36.268235 2003 ts_tablet_manager.cc:1939] T 1a8a178bafcb457f9b0e9587df560493 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:36.268464 1177 tablet_bootstrap.cc:654] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.271010 2024 rpcz_store.cc:276] 0112 02:52:34.415941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.416177 (+ 236us) service_pool.cc:225] Handling call
0112 02:52:35.789806 (+1373629us) ts_tablet_manager.cc:1145] Deleting tablet 1a1e3af2883341e9b957f6ed748e604d
0112 02:52:35.789819 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.221473 (+431654us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.224724 (+ 3251us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.260732 (+ 36008us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15605,"tablet-delete.queue_time_us":1373057}
I20250112 02:52:36.271391 2024 tablet_replica.cc:331] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:36.271638 2024 raft_consensus.cc:2238] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:36.271791 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dda52f012794493a102ed1abc718f5b" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:36.271935 1053 raft_consensus.cc:3054] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.272074 2024 raft_consensus.cc:2267] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.272647 2024 ts_tablet_manager.cc:1905] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.274156 1053 raft_consensus.cc:2463] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:36.275684 2217 raft_consensus.cc:491] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.275883 2217 raft_consensus.cc:513] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.280957 2215 raft_consensus.cc:2798] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
W20250112 02:52:36.281020 989 consensus_peers.cc:487] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73136be25c79491784d3ea71aa9b27f2. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:36.281106 2215 raft_consensus.cc:695] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:36.281375 2215 consensus_queue.cc:237] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.282680 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 1a1e3af2883341e9b957f6ed748e604d (table default.Table_b3ed6c7a79ed43568b8743f8aea9f67e [id=902e40c3f1e9401e934a4a2652f3dc22]) successfully deleted
I20250112 02:52:36.287608 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc414d6420c41839621ab858067548b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:36.287802 1053 raft_consensus.cc:2463] T 5cc414d6420c41839621ab858067548b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
W20250112 02:52:36.291992 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 56696.4us
I20250112 02:52:36.294862 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ee98f055024429eafaa29bb961cab87" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:36.295037 780 raft_consensus.cc:3054] T 8ee98f055024429eafaa29bb961cab87 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.295780 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6117dc8b32a64535a2f4167a03bed0d2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:36.295904 773 raft_consensus.cc:3054] T 6117dc8b32a64535a2f4167a03bed0d2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.297544 780 raft_consensus.cc:2463] T 8ee98f055024429eafaa29bb961cab87 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:36.298215 773 raft_consensus.cc:2463] T 6117dc8b32a64535a2f4167a03bed0d2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:36.300680 1173 tablet_bootstrap.cc:492] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:36.300815 1173 ts_tablet_manager.cc:1397] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20250112 02:52:36.302220 1173 raft_consensus.cc:357] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.302435 1173 raft_consensus.cc:383] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.302485 1173 raft_consensus.cc:738] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.302677 1173 consensus_queue.cc:260] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.303901 1173 ts_tablet_manager.cc:1428] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:36.304195 1173 tablet_bootstrap.cc:492] T ac716c6da01941a989754aac1ef430d5 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:36.309587 1173 tablet_bootstrap.cc:654] T ac716c6da01941a989754aac1ef430d5 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.318931 2024 ts_tablet_manager.cc:1918] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.319036 2024 log.cc:1198] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/c7c530aa0aa64e47ae27d61f341d2bd9
I20250112 02:52:36.319387 2024 ts_tablet_manager.cc:1939] T c7c530aa0aa64e47ae27d61f341d2bd9 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:36.320115 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 38) took 1903 ms. Trace:
I20250112 02:52:36.329063 2024 rpcz_store.cc:276] 0112 02:52:34.416462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.417420 (+ 958us) service_pool.cc:225] Handling call
0112 02:52:36.271353 (+1853933us) ts_tablet_manager.cc:1145] Deleting tablet c7c530aa0aa64e47ae27d61f341d2bd9
0112 02:52:36.271361 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.275213 (+ 3852us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.318859 (+ 43646us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.320098 (+ 1239us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1853663}
I20250112 02:52:36.329456 2024 tablet_replica.cc:331] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:36.330504 1177 tablet_bootstrap.cc:492] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:36.330154 1173 tablet_bootstrap.cc:492] T ac716c6da01941a989754aac1ef430d5 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:36.330660 1177 ts_tablet_manager.cc:1397] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.065s user 0.001s sys 0.007s
I20250112 02:52:36.330706 1173 ts_tablet_manager.cc:1397] T ac716c6da01941a989754aac1ef430d5 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.001s
I20250112 02:52:36.331395 1173 raft_consensus.cc:357] T ac716c6da01941a989754aac1ef430d5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.331496 1177 raft_consensus.cc:357] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.331702 1177 raft_consensus.cc:383] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.331761 1177 raft_consensus.cc:738] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.331746 1173 raft_consensus.cc:383] T ac716c6da01941a989754aac1ef430d5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.331979 1177 consensus_queue.cc:260] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.332448 1177 ts_tablet_manager.cc:1428] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:36.332665 1177 tablet_bootstrap.cc:492] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:36.332090 1173 raft_consensus.cc:738] T ac716c6da01941a989754aac1ef430d5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.334504 1177 tablet_bootstrap.cc:654] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.334698 1173 consensus_queue.cc:260] T ac716c6da01941a989754aac1ef430d5 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.275686 2036 ts_tablet_manager.cc:1918] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:36.284206 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ee98f055024429eafaa29bb961cab87" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:36.335034 2036 log.cc:1198] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/ac716c6da01941a989754aac1ef430d5
I20250112 02:52:36.335100 917 raft_consensus.cc:3054] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.334992 1181 raft_consensus.cc:383] T 781fe3514f4c4f80b1ee49278502ccc1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.335243 1181 raft_consensus.cc:738] T 781fe3514f4c4f80b1ee49278502ccc1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.335367 2036 ts_tablet_manager.cc:1939] T ac716c6da01941a989754aac1ef430d5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:36.336047 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 51) took 2127 ms. Trace:
I20250112 02:52:36.336113 2036 rpcz_store.cc:276] 0112 02:52:34.208659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.277026 (+ 68367us) service_pool.cc:225] Handling call
0112 02:52:36.199726 (+1922700us) ts_tablet_manager.cc:1145] Deleting tablet ac716c6da01941a989754aac1ef430d5
0112 02:52:36.199735 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.226613 (+ 26878us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.275605 (+ 48992us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.336031 (+ 60426us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1922570}
I20250112 02:52:36.336465 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 52) took 2115 ms. Trace:
I20250112 02:52:36.336532 2036 rpcz_store.cc:276] 0112 02:52:34.221147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.277144 (+ 55997us) service_pool.cc:225] Handling call
0112 02:52:36.336338 (+2059194us) ts_tablet_manager.cc:1145] Deleting tablet bab4978edea0445fb0456e88e7fb4c44
0112 02:52:36.336346 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.336454 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2059085}
I20250112 02:52:36.336789 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 53) took 2106 ms. Trace:
I20250112 02:52:36.336843 2036 rpcz_store.cc:276] 0112 02:52:34.229860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.277158 (+ 47298us) service_pool.cc:225] Handling call
0112 02:52:36.336709 (+2059551us) ts_tablet_manager.cc:1145] Deleting tablet 058a9ca089cd4967887b0a1127c7d03f
0112 02:52:36.336716 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.336776 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2059448}
I20250112 02:52:36.337054 2036 tablet_replica.cc:331] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:36.337253 2036 raft_consensus.cc:2238] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.337383 2036 raft_consensus.cc:2267] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.337818 2036 ts_tablet_manager.cc:1905] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.338268 917 raft_consensus.cc:2463] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:36.338603 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet ac716c6da01941a989754aac1ef430d5 (table default.Table_bee5a719ea4b44b78254bae566a0ba92 [id=d3fa1ea0552e4aaeabd6ac3ac84ca389]) successfully deleted
W20250112 02:52:36.338850 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet bab4978edea0445fb0456e88e7fb4c44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bab4978edea0445fb0456e88e7fb4c44 already in progress: creating tablet
W20250112 02:52:36.339067 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 058a9ca089cd4967887b0a1127c7d03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 058a9ca089cd4967887b0a1127c7d03f already in progress: creating tablet
I20250112 02:52:36.345124 2036 ts_tablet_manager.cc:1918] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:36.345273 2036 log.cc:1198] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/b8daf4ef3593410eae87672402292be6
I20250112 02:52:36.345476 1177 tablet_bootstrap.cc:492] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:36.345597 1177 ts_tablet_manager.cc:1397] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.003s
I20250112 02:52:36.345613 2036 ts_tablet_manager.cc:1939] T b8daf4ef3593410eae87672402292be6 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:36.346261 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 54) took 2116 ms. Trace:
I20250112 02:52:36.346235 1177 raft_consensus.cc:357] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.346408 1177 raft_consensus.cc:383] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.346483 1177 raft_consensus.cc:738] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.346534 2024 raft_consensus.cc:2238] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:36.346740 2036 rpcz_store.cc:276] 0112 02:52:34.230135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.277261 (+ 47126us) service_pool.cc:225] Handling call
0112 02:52:36.337022 (+2059761us) ts_tablet_manager.cc:1145] Deleting tablet b8daf4ef3593410eae87672402292be6
0112 02:52:36.337028 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.341179 (+ 4151us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.345061 (+ 3882us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.346243 (+ 1182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2059631}
I20250112 02:52:36.348282 1177 consensus_queue.cc:260] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.348682 2036 tablet_replica.cc:331] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:36.348798 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet b8daf4ef3593410eae87672402292be6 (table default.Table_c0aaadb5e69e46e8983f4d289a44afe2 [id=08299ab225a94e37a67722107a5ff4ce]) successfully deleted
I20250112 02:52:36.349018 2208 raft_consensus.cc:491] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.348960 2036 raft_consensus.cc:2238] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:36.349117 2208 raft_consensus.cc:513] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.349694 1173 ts_tablet_manager.cc:1428] T ac716c6da01941a989754aac1ef430d5 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.019s user 0.001s sys 0.000s
I20250112 02:52:36.350055 1173 tablet_bootstrap.cc:492] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:36.350090 1177 ts_tablet_manager.cc:1428] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 02:52:36.350642 2024 raft_consensus.cc:2267] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.351358 1177 tablet_bootstrap.cc:492] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:36.351548 2036 raft_consensus.cc:2267] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.352171 2036 ts_tablet_manager.cc:1905] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.353437 1177 tablet_bootstrap.cc:654] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.354025 449 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 127) took 1671 ms. Trace:
I20250112 02:52:36.354209 2217 leader_election.cc:290] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:36.354344 2208 leader_election.cc:290] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:36.354136 449 rpcz_store.cc:276] 0112 02:52:34.682604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:35.724561 (+1041957us) service_pool.cc:225] Handling call
0112 02:52:35.725516 (+ 955us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:35.726047 (+ 531us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:36.353704 (+627657us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:36.353894 (+ 190us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:36.353991 (+ 97us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:35.858313 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:35.858468 (+ 155us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:35.858477 (+ 9us) write_op.cc:435] Acquired schema lock
0112 02:52:35.858481 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:35.858600 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:35.858617 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:35.858625 (+ 8us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:35.858717 (+ 92us) tablet.cc:698] Row locks acquired
0112 02:52:35.858721 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:35.858758 (+ 37us) write_op.cc:270] Start()
0112 02:52:35.858797 (+ 39us) write_op.cc:276] Timestamp: P: 1736650355858676 usec, L: 0
0112 02:52:35.858802 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:35.859014 (+ 212us) log.cc:844] Serialized 433 byte log entry
0112 02:52:36.324032 (+465018us) op_driver.cc:464] REPLICATION: finished
0112 02:52:36.352088 (+ 28056us) write_op.cc:301] APPLY: starting
0112 02:52:36.352104 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:36.352139 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:36.352190 (+ 51us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:36.352328 (+ 138us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:36.352372 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:36.352378 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:36.352465 (+ 87us) log.cc:844] Serialized 29 byte log entry
0112 02:52:36.353592 (+ 1127us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.353612 (+ 20us) write_op.cc:454] Released schema lock
0112 02:52:36.353619 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:36.353700 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.353704 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":27990,"apply.run_cpu_time_us":581,"apply.run_wall_time_us":1612,"mutex_wait_us":299356,"num_ops":2,"prepare.queue_time_us":131696,"prepare.run_cpu_time_us":938,"prepare.run_wall_time_us":300695,"raft.queue_time_us":299498,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":269,"replication_time_us":465184,"thread_start_us":161494,"threads_started":2}]]}
I20250112 02:52:36.358021 2179 raft_consensus.cc:491] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.358122 2179 raft_consensus.cc:513] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.358680 2179 leader_election.cc:290] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:36.359232 2024 ts_tablet_manager.cc:1905] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.361753 1177 tablet_bootstrap.cc:492] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:36.361867 1177 ts_tablet_manager.cc:1397] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250112 02:52:36.362479 1177 raft_consensus.cc:357] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.362633 1177 raft_consensus.cc:383] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.362702 1177 raft_consensus.cc:738] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
W20250112 02:52:36.362833 713 leader_election.cc:343] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: ac716c6da01941a989754aac1ef430d5
I20250112 02:52:36.364090 1177 consensus_queue.cc:260] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.366178 1177 ts_tablet_manager.cc:1428] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 02:52:36.335441 1181 consensus_queue.cc:260] T 781fe3514f4c4f80b1ee49278502ccc1 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.366953 1181 ts_tablet_manager.cc:1428] T 781fe3514f4c4f80b1ee49278502ccc1 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.101s user 0.001s sys 0.000s
I20250112 02:52:36.367798 1181 tablet_bootstrap.cc:492] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:36.369982 1181 tablet_bootstrap.cc:654] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.370554 1177 tablet_bootstrap.cc:492] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:36.372247 1177 tablet_bootstrap.cc:654] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.376349 771 raft_consensus.cc:3054] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.386620 2179 raft_consensus.cc:491] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.386734 2179 raft_consensus.cc:513] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.268926 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 44) took 2145 ms. Trace:
I20250112 02:52:36.388080 2003 rpcz_store.cc:276] 0112 02:52:34.123169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.123225 (+ 56us) service_pool.cc:225] Handling call
0112 02:52:36.239694 (+2116469us) ts_tablet_manager.cc:1145] Deleting tablet 1a8a178bafcb457f9b0e9587df560493
0112 02:52:36.239703 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.254344 (+ 14641us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.267703 (+ 13359us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.268911 (+ 1208us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5199,"tablet-delete.queue_time_us":2113562}
I20250112 02:52:36.390559 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 1a8a178bafcb457f9b0e9587df560493 (table default.Table_3c1f0242d17443b5a758967cf82ca0ca [id=02303baf5298474488dbf02c831f3d4d]) successfully deleted
I20250112 02:52:36.390693 714 leader_election.cc:304] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:36.391275 2190 raft_consensus.cc:2798] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:36.391373 2190 raft_consensus.cc:695] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:36.391855 2003 tablet_replica.cc:331] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:36.392076 2003 raft_consensus.cc:2238] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.392683 2179 leader_election.cc:290] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:36.392820 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac716c6da01941a989754aac1ef430d5" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:36.392998 1051 raft_consensus.cc:2463] T ac716c6da01941a989754aac1ef430d5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:36.393246 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d25815a5df94e099e7481c48c3da793" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:36.399361 1181 tablet_bootstrap.cc:492] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:36.401635 2228 consensus_queue.cc:1035] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:36.402729 1181 ts_tablet_manager.cc:1397] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20250112 02:52:36.403715 1181 raft_consensus.cc:357] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:36.410470 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 52978.6us
I20250112 02:52:36.411522 2003 raft_consensus.cc:2267] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.412072 2003 ts_tablet_manager.cc:1905] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.404431 854 leader_election.cc:304] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:36.413298 2222 raft_consensus.cc:2798] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:36.413372 2222 raft_consensus.cc:491] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:36.413430 2222 raft_consensus.cc:3054] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.413792 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31b777e7e0f43a28839d8c48c6a6590" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:36.413967 776 raft_consensus.cc:2463] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:36.414191 1181 raft_consensus.cc:383] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.414283 1181 raft_consensus.cc:738] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.414477 1181 consensus_queue.cc:260] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.416056 446 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 131) took 1579 ms. Trace:
I20250112 02:52:36.416126 1181 ts_tablet_manager.cc:1428] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20250112 02:52:36.416388 1181 tablet_bootstrap.cc:492] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:36.416162 446 rpcz_store.cc:276] 0112 02:52:34.836986 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:35.745283 (+908297us) service_pool.cc:225] Handling call
0112 02:52:35.746080 (+ 797us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:35.746515 (+ 435us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:36.415822 (+669307us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:36.415922 (+ 100us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:36.416003 (+ 81us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:36.228580 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:36.228703 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:36.228710 (+ 7us) write_op.cc:435] Acquired schema lock
0112 02:52:36.228712 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:36.228800 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:36.228810 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:36.228814 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:36.228883 (+ 69us) tablet.cc:698] Row locks acquired
0112 02:52:36.228885 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:36.228914 (+ 29us) write_op.cc:270] Start()
0112 02:52:36.228942 (+ 28us) write_op.cc:276] Timestamp: P: 1736650356228862 usec, L: 0
0112 02:52:36.228946 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:36.229119 (+ 173us) log.cc:844] Serialized 433 byte log entry
0112 02:52:36.415056 (+185937us) op_driver.cc:464] REPLICATION: finished
0112 02:52:36.415270 (+ 214us) write_op.cc:301] APPLY: starting
0112 02:52:36.415281 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:36.415311 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:36.415332 (+ 21us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:36.415488 (+ 156us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:36.415512 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:36.415522 (+ 10us) write_op.cc:312] APPLY: finished
0112 02:52:36.415589 (+ 67us) log.cc:844] Serialized 29 byte log entry
0112 02:52:36.415662 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.415681 (+ 19us) write_op.cc:454] Released schema lock
0112 02:52:36.415689 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":178,"apply.run_cpu_time_us":480,"apply.run_wall_time_us":481,"num_ops":2,"prepare.queue_time_us":481633,"prepare.run_cpu_time_us":675,"prepare.run_wall_time_us":77444,"raft.queue_time_us":175,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":186085}]]}
I20250112 02:52:36.391579 2190 consensus_queue.cc:237] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.422984 2222 raft_consensus.cc:513] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:36.423144 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d25815a5df94e099e7481c48c3da793" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:36.423372 918 raft_consensus.cc:2463] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:36.423653 2222 leader_election.cc:290] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:36.423756 713 leader_election.cc:304] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:36.424005 2179 raft_consensus.cc:2798] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:36.424075 2179 raft_consensus.cc:491] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:36.424008 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc414d6420c41839621ab858067548b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:36.424127 2179 raft_consensus.cc:3054] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.424177 776 raft_consensus.cc:3054] T 5cc414d6420c41839621ab858067548b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.426713 2036 ts_tablet_manager.cc:1918] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.426803 2024 ts_tablet_manager.cc:1918] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.428056 2036 log.cc:1198] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/64707ee432fb4e96a3bbf7252dfa1813
I20250112 02:52:36.429251 1181 tablet_bootstrap.cc:654] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.429467 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet c7c530aa0aa64e47ae27d61f341d2bd9 (table default.Table_247a821819994de6adac6fabd06c3a1f [id=60486b9e81254d74b8e08a1cadde13bd]) successfully deleted
I20250112 02:52:36.428426 2036 ts_tablet_manager.cc:1939] T 64707ee432fb4e96a3bbf7252dfa1813 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:36.430893 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 55) took 2191 ms. Trace:
I20250112 02:52:36.430979 2036 rpcz_store.cc:276] 0112 02:52:34.239230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.277266 (+ 38036us) service_pool.cc:225] Handling call
0112 02:52:36.348647 (+2071381us) ts_tablet_manager.cc:1145] Deleting tablet 64707ee432fb4e96a3bbf7252dfa1813
0112 02:52:36.348655 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.374126 (+ 25471us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.426619 (+ 52493us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.430877 (+ 4258us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2071263}
I20250112 02:52:36.431299 2036 tablet_replica.cc:331] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:36.431520 2036 raft_consensus.cc:2238] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.431488 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 64707ee432fb4e96a3bbf7252dfa1813 (table default.Table_c26869f6cbd540fe980dcf207b5c729d [id=cdd6a2097792484d84bca856f30d7ac1]) successfully deleted
I20250112 02:52:36.431674 2036 raft_consensus.cc:2267] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.432173 2036 ts_tablet_manager.cc:1905] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.432176 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31b777e7e0f43a28839d8c48c6a6590" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:36.432514 916 raft_consensus.cc:2463] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:36.439502 2179 raft_consensus.cc:513] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.440146 2179 leader_election.cc:290] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:36.440663 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d25815a5df94e099e7481c48c3da793" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:36.440879 918 raft_consensus.cc:3054] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.442422 2003 ts_tablet_manager.cc:1918] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.442528 2003 log.cc:1198] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/3c686fdc8896463c81c9f7c5f53a3da2
I20250112 02:52:36.442718 776 raft_consensus.cc:2463] T 5cc414d6420c41839621ab858067548b P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:36.442885 2003 ts_tablet_manager.cc:1939] T 3c686fdc8896463c81c9f7c5f53a3da2 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:36.443125 855 leader_election.cc:304] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:36.443313 2222 raft_consensus.cc:2798] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:36.443713 2222 raft_consensus.cc:695] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:36.443989 2222 consensus_queue.cc:237] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:36.444927 918 raft_consensus.cc:2463] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:36.445318 713 leader_election.cc:304] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:36.452132 2179 raft_consensus.cc:2798] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:36.452286 2179 raft_consensus.cc:695] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:36.452526 2179 consensus_queue.cc:237] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.458348 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 45) took 2334 ms. Trace:
I20250112 02:52:36.458477 2003 rpcz_store.cc:276] 0112 02:52:34.123698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.123760 (+ 62us) service_pool.cc:225] Handling call
0112 02:52:36.388398 (+2264638us) ts_tablet_manager.cc:1145] Deleting tablet 3c686fdc8896463c81c9f7c5f53a3da2
0112 02:52:36.388407 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.436164 (+ 47757us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.442356 (+ 6192us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.458332 (+ 15976us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2255418}
I20250112 02:52:36.458792 2003 tablet_replica.cc:331] T 49bb6d20cfa0453e90e048825114525e P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:36.458976 2003 raft_consensus.cc:2238] T 49bb6d20cfa0453e90e048825114525e P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.459374 2003 raft_consensus.cc:2267] T 49bb6d20cfa0453e90e048825114525e P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.460201 2003 ts_tablet_manager.cc:1905] T 49bb6d20cfa0453e90e048825114525e P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.462742 447 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 132) took 1610 ms. Trace:
I20250112 02:52:36.462850 447 rpcz_store.cc:276] 0112 02:52:34.852735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:35.745981 (+893246us) service_pool.cc:225] Handling call
0112 02:52:35.746753 (+ 772us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:35.747266 (+ 513us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:36.460793 (+713527us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:36.462635 (+ 1842us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:36.462711 (+ 76us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:36.306054 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:36.306163 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:36.306171 (+ 8us) write_op.cc:435] Acquired schema lock
0112 02:52:36.306173 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:36.306264 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:36.306274 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:36.306279 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:36.306361 (+ 82us) tablet.cc:698] Row locks acquired
0112 02:52:36.306365 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:36.306404 (+ 39us) write_op.cc:270] Start()
0112 02:52:36.306443 (+ 39us) write_op.cc:276] Timestamp: P: 1736650356306336 usec, L: 0
0112 02:52:36.306447 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:36.306606 (+ 159us) log.cc:844] Serialized 433 byte log entry
0112 02:52:36.460264 (+153658us) op_driver.cc:464] REPLICATION: finished
0112 02:52:36.460335 (+ 71us) write_op.cc:301] APPLY: starting
0112 02:52:36.460348 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:36.460383 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:36.460421 (+ 38us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:36.460509 (+ 88us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:36.460536 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:36.460543 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:36.460599 (+ 56us) log.cc:844] Serialized 29 byte log entry
0112 02:52:36.460654 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.460672 (+ 18us) write_op.cc:454] Released schema lock
0112 02:52:36.460678 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":32,"apply.run_cpu_time_us":396,"apply.run_wall_time_us":396,"num_ops":2,"prepare.queue_time_us":558347,"prepare.run_cpu_time_us":667,"prepare.run_wall_time_us":21877,"raft.queue_time_us":34,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":153788}]]}
I20250112 02:52:36.437070 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc414d6420c41839621ab858067548b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:36.469259 2024 log.cc:1198] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/4c0c2fad4f3243238641aae56b305bb5
W20250112 02:52:36.444636 989 consensus_peers.cc:487] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb217afc08304c0ea35dee3120347ba1. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:36.447885 1173 tablet_bootstrap.cc:654] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.469928 2024 ts_tablet_manager.cc:1939] T 4c0c2fad4f3243238641aae56b305bb5 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:36.470320 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 39) took 2052 ms. Trace:
I20250112 02:52:36.469295 1051 raft_consensus.cc:3054] T 5cc414d6420c41839621ab858067548b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:36.470811 989 consensus_peers.cc:487] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b599e983ec04f53a440d1ab828d12ab. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:52:36.437536 991 consensus_peers.cc:487] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb217afc08304c0ea35dee3120347ba1. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:36.477989 2228 consensus_queue.cc:1035] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:36.478343 1051 raft_consensus.cc:2463] T 5cc414d6420c41839621ab858067548b P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:36.478500 989 leader_election.cc:304] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:36.478879 2208 raft_consensus.cc:2798] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:36.478974 989 leader_election.cc:304] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:36.479069 2207 consensus_queue.cc:1035] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:36.479408 2207 raft_consensus.cc:2798] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:36.479471 2207 raft_consensus.cc:491] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:36.479553 2207 raft_consensus.cc:3054] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.479669 2216 tablet.cc:1721] T 48634d6d78784fae98512b242e5476df P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:36.478976 2208 raft_consensus.cc:695] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:36.480232 2047 raft_consensus.cc:491] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.480327 2208 consensus_queue.cc:237] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.480351 2047 raft_consensus.cc:513] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.480949 2047 leader_election.cc:290] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:36.481848 2207 raft_consensus.cc:513] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.482566 2207 leader_election.cc:290] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:36.485177 991 consensus_peers.cc:487] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65f22a7b1b804661af3b7e785a833739. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:52:36.486114 991 consensus_peers.cc:487] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35f59617b34046568dd83bbb7cdc42d2. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:36.486776 2024 rpcz_store.cc:276] 0112 02:52:34.418184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.418714 (+ 530us) service_pool.cc:225] Handling call
0112 02:52:36.329416 (+1910702us) ts_tablet_manager.cc:1145] Deleting tablet 4c0c2fad4f3243238641aae56b305bb5
0112 02:52:36.329424 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.390242 (+ 60818us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.426719 (+ 36477us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.470307 (+ 43588us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3850,"tablet-delete.queue_time_us":1910512}
I20250112 02:52:36.487360 2024 tablet_replica.cc:331] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:36.488965 2024 raft_consensus.cc:2238] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:36.489454 1177 tablet_bootstrap.cc:492] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:36.489562 1177 ts_tablet_manager.cc:1397] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.123s user 0.007s sys 0.007s
W20250112 02:52:36.489737 991 consensus_peers.cc:487] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42cb7e7a41634cf1bdb978ac9d2befcd. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:36.489818 2024 raft_consensus.cc:2267] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.490177 1177 raft_consensus.cc:357] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.490403 1177 raft_consensus.cc:383] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.490463 1177 raft_consensus.cc:738] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.491236 2024 ts_tablet_manager.cc:1905] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.495761 1177 consensus_queue.cc:260] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.496215 1177 ts_tablet_manager.cc:1428] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250112 02:52:36.500774 2003 ts_tablet_manager.cc:1918] T 49bb6d20cfa0453e90e048825114525e P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.502957 2024 ts_tablet_manager.cc:1918] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.503315 2024 log.cc:1198] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/aadb2ab1d2024f889841ac91e065e064
I20250112 02:52:36.503715 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e933b58dab0b45e8bf1c49eebb1c53e3" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:36.504037 918 raft_consensus.cc:2463] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:36.504648 2003 log.cc:1198] T 49bb6d20cfa0453e90e048825114525e P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/49bb6d20cfa0453e90e048825114525e
I20250112 02:52:36.504981 2003 ts_tablet_manager.cc:1939] T 49bb6d20cfa0453e90e048825114525e P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:36.505604 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 47) took 2373 ms. Trace:
I20250112 02:52:36.505682 2003 rpcz_store.cc:276] 0112 02:52:34.131699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.131740 (+ 41us) service_pool.cc:225] Handling call
0112 02:52:36.458747 (+2327007us) ts_tablet_manager.cc:1145] Deleting tablet 49bb6d20cfa0453e90e048825114525e
0112 02:52:36.458756 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.477581 (+ 18825us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.500706 (+ 23125us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.505587 (+ 4881us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21122,"tablet-delete.queue_time_us":2309043}
I20250112 02:52:36.505859 2024 ts_tablet_manager.cc:1939] T aadb2ab1d2024f889841ac91e065e064 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:36.505954 2003 tablet_replica.cc:331] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:36.506114 2003 raft_consensus.cc:2238] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:36.506364 2003 raft_consensus.cc:2267] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.503500 1177 tablet_bootstrap.cc:492] T 87759e8b99b04e628badc8f60cb38bb3 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:36.504447 713 leader_election.cc:304] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:36.507122 2047 raft_consensus.cc:2798] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:36.507244 2047 raft_consensus.cc:491] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:36.507301 2047 raft_consensus.cc:3054] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.507426 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 40) took 2088 ms. Trace:
I20250112 02:52:36.507613 2024 rpcz_store.cc:276] 0112 02:52:34.419096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.420005 (+ 909us) service_pool.cc:225] Handling call
0112 02:52:36.487303 (+2067298us) ts_tablet_manager.cc:1145] Deleting tablet aadb2ab1d2024f889841ac91e065e064
0112 02:52:36.487315 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.496123 (+ 8808us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.502879 (+ 6756us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.507382 (+ 4503us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":250,"tablet-delete.queue_time_us":2065682}
I20250112 02:52:36.508399 2024 tablet_replica.cc:331] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:36.508738 1177 tablet_bootstrap.cc:654] T 87759e8b99b04e628badc8f60cb38bb3 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.508883 2024 raft_consensus.cc:2238] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.509080 2024 raft_consensus.cc:2267] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.509954 2047 raft_consensus.cc:513] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.510612 2047 leader_election.cc:290] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:36.510730 2024 ts_tablet_manager.cc:1905] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.510972 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e933b58dab0b45e8bf1c49eebb1c53e3" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:36.511215 918 raft_consensus.cc:3054] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.511942 1181 tablet_bootstrap.cc:492] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:36.512089 1181 ts_tablet_manager.cc:1397] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.096s user 0.003s sys 0.000s
I20250112 02:52:36.512950 1181 raft_consensus.cc:357] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.513163 1181 raft_consensus.cc:383] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.513231 1181 raft_consensus.cc:738] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.513458 1181 consensus_queue.cc:260] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.514082 1181 ts_tablet_manager.cc:1428] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:36.514355 1181 tablet_bootstrap.cc:492] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:36.515686 1181 tablet_bootstrap.cc:654] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.518806 918 raft_consensus.cc:2463] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:36.519230 713 leader_election.cc:304] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:36.519546 2047 raft_consensus.cc:2798] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:36.519697 2047 raft_consensus.cc:695] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:36.520614 2047 consensus_queue.cc:237] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:36.490207 991 leader_election.cc:343] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: e9fc9a4f1e8944abb76dad6c2ad53d35
I20250112 02:52:36.521714 2208 consensus_queue.cc:1035] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:36.523001 2024 ts_tablet_manager.cc:1918] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.523245 2024 log.cc:1198] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/41e166b3f8414ababbd4a3f07fb90c4c
I20250112 02:52:36.523815 2024 ts_tablet_manager.cc:1939] T 41e166b3f8414ababbd4a3f07fb90c4c P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:36.524911 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 42) took 2104 ms. Trace:
I20250112 02:52:36.525023 2024 rpcz_store.cc:276] 0112 02:52:34.420573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.420616 (+ 43us) service_pool.cc:225] Handling call
0112 02:52:36.508176 (+2087560us) ts_tablet_manager.cc:1145] Deleting tablet 41e166b3f8414ababbd4a3f07fb90c4c
0112 02:52:36.508186 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.520247 (+ 12061us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.522935 (+ 2688us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.524894 (+ 1959us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2086308}
I20250112 02:52:36.525681 2024 tablet_replica.cc:331] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:36.525884 2024 raft_consensus.cc:2238] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.526026 2024 raft_consensus.cc:2267] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.526610 2024 ts_tablet_manager.cc:1905] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.527338 2036 ts_tablet_manager.cc:1918] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.528663 2003 ts_tablet_manager.cc:1905] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.535754 1181 tablet_bootstrap.cc:492] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:36.535925 1181 ts_tablet_manager.cc:1397] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20250112 02:52:36.536684 1181 raft_consensus.cc:357] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.536890 1181 raft_consensus.cc:383] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.536983 1181 raft_consensus.cc:738] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.537302 1181 consensus_queue.cc:260] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.537910 1181 ts_tablet_manager.cc:1428] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:36.538174 1181 tablet_bootstrap.cc:492] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:36.540086 1181 tablet_bootstrap.cc:654] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.540442 2024 ts_tablet_manager.cc:1918] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:36.540900 2024 log.cc:1198] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/a47cdc208fbc431b8326364e84432cf5
I20250112 02:52:36.541330 2024 ts_tablet_manager.cc:1939] T a47cdc208fbc431b8326364e84432cf5 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:36.543568 1177 tablet_bootstrap.cc:492] T 87759e8b99b04e628badc8f60cb38bb3 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:36.543701 1177 ts_tablet_manager.cc:1397] T 87759e8b99b04e628badc8f60cb38bb3 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20250112 02:52:36.544055 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 43) took 2123 ms. Trace:
I20250112 02:52:36.544204 2024 rpcz_store.cc:276] 0112 02:52:34.420877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.420925 (+ 48us) service_pool.cc:225] Handling call
0112 02:52:36.525458 (+2104533us) ts_tablet_manager.cc:1145] Deleting tablet a47cdc208fbc431b8326364e84432cf5
0112 02:52:36.525651 (+ 193us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.530631 (+ 4980us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.533039 (+ 2408us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.544040 (+ 11001us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2098620}
I20250112 02:52:36.544618 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 44) took 2123 ms. Trace:
I20250112 02:52:36.544754 2024 rpcz_store.cc:276] 0112 02:52:34.421219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.421271 (+ 52us) service_pool.cc:225] Handling call
0112 02:52:36.544488 (+2123217us) ts_tablet_manager.cc:1145] Deleting tablet fe614eaeb793473d9b2da206b7345562
0112 02:52:36.544498 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.544601 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2116918}
I20250112 02:52:36.545238 2024 tablet_replica.cc:331] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:36.545487 2024 raft_consensus.cc:2238] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.545678 2024 raft_consensus.cc:2267] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.546167 2024 ts_tablet_manager.cc:1905] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.549801 1181 tablet_bootstrap.cc:492] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:36.549268 2332 tablet.cc:1721] T 48634d6d78784fae98512b242e5476df P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:36.549939 1181 ts_tablet_manager.cc:1397] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20250112 02:52:36.550505 1181 raft_consensus.cc:357] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:36.550655 1181 raft_consensus.cc:383] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.550700 1181 raft_consensus.cc:738] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.550850 1181 consensus_queue.cc:260] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:36.551283 1181 ts_tablet_manager.cc:1428] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:36.551530 1181 tablet_bootstrap.cc:492] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:36.551692 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31b777e7e0f43a28839d8c48c6a6590" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:36.551906 775 raft_consensus.cc:3054] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.552948 1181 tablet_bootstrap.cc:654] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.553932 1177 raft_consensus.cc:357] T 87759e8b99b04e628badc8f60cb38bb3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.555199 775 raft_consensus.cc:2463] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:36.555506 1177 raft_consensus.cc:383] T 87759e8b99b04e628badc8f60cb38bb3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.555595 1177 raft_consensus.cc:738] T 87759e8b99b04e628badc8f60cb38bb3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.556907 2024 ts_tablet_manager.cc:1918] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:36.557261 2024 log.cc:1198] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/a1a0a13696fc41ccb28218f1275a4b5d
I20250112 02:52:36.557606 2024 ts_tablet_manager.cc:1939] T a1a0a13696fc41ccb28218f1275a4b5d P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:36.556608 1177 consensus_queue.cc:260] T 87759e8b99b04e628badc8f60cb38bb3 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.558454 1177 ts_tablet_manager.cc:1428] T 87759e8b99b04e628badc8f60cb38bb3 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250112 02:52:36.558964 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 41) took 2138 ms. Trace:
I20250112 02:52:36.559481 1181 tablet_bootstrap.cc:492] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:36.559623 1181 ts_tablet_manager.cc:1397] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20250112 02:52:36.560266 1181 raft_consensus.cc:357] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.560452 1181 raft_consensus.cc:383] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.560554 1181 raft_consensus.cc:738] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.560741 1181 consensus_queue.cc:260] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.561259 1181 ts_tablet_manager.cc:1428] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 02:52:36.561512 1181 tablet_bootstrap.cc:492] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:36.563772 2024 rpcz_store.cc:276] 0112 02:52:34.420301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.420347 (+ 46us) service_pool.cc:225] Handling call
0112 02:52:36.545191 (+2124844us) ts_tablet_manager.cc:1145] Deleting tablet a1a0a13696fc41ccb28218f1275a4b5d
0112 02:52:36.545203 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.554218 (+ 9015us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.556848 (+ 2630us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.558948 (+ 2100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2117480}
I20250112 02:52:36.563970 1177 tablet_bootstrap.cc:492] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:36.564106 2024 tablet_replica.cc:331] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:36.564270 2024 raft_consensus.cc:2238] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.564448 2024 raft_consensus.cc:2267] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.564589 1181 tablet_bootstrap.cc:654] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.565027 2024 ts_tablet_manager.cc:1905] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:36.568831 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 89859.1us
W20250112 02:52:36.569031 989 leader_election.cc:343] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: e9fc9a4f1e8944abb76dad6c2ad53d35
I20250112 02:52:36.569106 2003 ts_tablet_manager.cc:1918] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.569140 989 leader_election.cc:304] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:36.569203 2003 log.cc:1198] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/61593969d2234ce3a6089027b067d922
I20250112 02:52:36.569520 2003 ts_tablet_manager.cc:1939] T 61593969d2234ce3a6089027b067d922 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
W20250112 02:52:36.569684 989 consensus_peers.cc:487] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea33e5c1a1fb46599c5836a36b130051. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:36.570147 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 49) took 2392 ms. Trace:
I20250112 02:52:36.570688 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31b777e7e0f43a28839d8c48c6a6590" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:36.570827 919 raft_consensus.cc:3054] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.571058 2215 raft_consensus.cc:2743] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:36.571894 2217 raft_consensus.cc:491] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.571995 2217 raft_consensus.cc:513] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.572250 452 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 133) took 1696 ms. Trace:
I20250112 02:52:36.572499 2217 leader_election.cc:290] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:36.572352 452 rpcz_store.cc:276] 0112 02:52:34.876038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:35.748049 (+872011us) service_pool.cc:225] Handling call
0112 02:52:35.748899 (+ 850us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:35.752604 (+ 3705us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:36.569351 (+816747us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:36.572088 (+ 2737us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:36.572194 (+ 106us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:36.386946 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:36.387054 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:36.387061 (+ 7us) write_op.cc:435] Acquired schema lock
0112 02:52:36.387063 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:36.387181 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:36.387192 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:36.387196 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:36.387281 (+ 85us) tablet.cc:698] Row locks acquired
0112 02:52:36.387283 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:36.387316 (+ 33us) write_op.cc:270] Start()
0112 02:52:36.387362 (+ 46us) write_op.cc:276] Timestamp: P: 1736650356387229 usec, L: 0
0112 02:52:36.387367 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:36.388089 (+ 722us) log.cc:844] Serialized 433 byte log entry
0112 02:52:36.568055 (+179966us) op_driver.cc:464] REPLICATION: finished
0112 02:52:36.568613 (+ 558us) write_op.cc:301] APPLY: starting
0112 02:52:36.568625 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:36.568658 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:36.569003 (+ 345us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:36.569094 (+ 91us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:36.569123 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:36.569131 (+ 8us) write_op.cc:312] APPLY: finished
0112 02:52:36.569192 (+ 61us) log.cc:844] Serialized 29 byte log entry
0112 02:52:36.569237 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.569252 (+ 15us) write_op.cc:454] Released schema lock
0112 02:52:36.569258 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":505,"apply.run_cpu_time_us":394,"apply.run_wall_time_us":700,"num_ops":2,"prepare.queue_time_us":633869,"prepare.run_cpu_time_us":7831,"prepare.run_wall_time_us":40449,"raft.queue_time_us":8032,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":7,"replication_time_us":180659,"thread_start_us":39182,"threads_started":2}]]}
I20250112 02:52:36.572938 2024 ts_tablet_manager.cc:1918] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:36.573019 2024 log.cc:1198] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/b2d8ea2ef18a46508371b4789ab6f4fa
I20250112 02:52:36.573235 919 raft_consensus.cc:2463] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:36.573282 2024 ts_tablet_manager.cc:1939] T b2d8ea2ef18a46508371b4789ab6f4fa P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:36.575541 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 45) took 2154 ms. Trace:
I20250112 02:52:36.576555 1181 tablet_bootstrap.cc:492] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:36.576725 1181 ts_tablet_manager.cc:1397] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20250112 02:52:36.577561 1181 raft_consensus.cc:357] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.577811 1181 raft_consensus.cc:383] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.577908 1181 raft_consensus.cc:738] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.578159 1181 consensus_queue.cc:260] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.578871 1181 ts_tablet_manager.cc:1428] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:36.579131 1181 tablet_bootstrap.cc:492] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:36.580822 1181 tablet_bootstrap.cc:654] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.580937 2024 rpcz_store.cc:276] 0112 02:52:34.421516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.431621 (+ 10105us) service_pool.cc:225] Handling call
0112 02:52:36.564074 (+2132453us) ts_tablet_manager.cc:1145] Deleting tablet b2d8ea2ef18a46508371b4789ab6f4fa
0112 02:52:36.564082 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.568708 (+ 4626us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.572859 (+ 4151us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.575521 (+ 2662us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1494,"tablet-delete.queue_time_us":2132236}
I20250112 02:52:36.581360 2024 tablet_replica.cc:331] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:36.581612 2024 raft_consensus.cc:2238] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:36.581967 2024 raft_consensus.cc:2267] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.582978 1177 tablet_bootstrap.cc:654] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.583379 2024 ts_tablet_manager.cc:1905] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.586676 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2730634a5cfd4c2fb5c4e30100c556d4" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:36.592675 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 4c0c2fad4f3243238641aae56b305bb5 (table default.Table_82662e02aee0459dad1b621df36f58f0 [id=50a3f0607e95430594c6769983f4e31b]) successfully deleted
I20250112 02:52:36.592988 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet aadb2ab1d2024f889841ac91e065e064 (table default.Table_6255caf5a65d4d9192e68f39413398f0 [id=5e4665095f4947bd8adb7ec66be4b456]) successfully deleted
I20250112 02:52:36.593186 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 41e166b3f8414ababbd4a3f07fb90c4c (table default.Table_11b4065cfb0449b68950d63e65f566a2 [id=0e325561bbdb468e8a64ca9808e2f3cf]) successfully deleted
I20250112 02:52:36.593361 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet a47cdc208fbc431b8326364e84432cf5 (table default.Table_bd1cdf8c9b3343af8d17a9fd05c77145 [id=c19f442c5dc7459389b6bac8beda390c]) successfully deleted
I20250112 02:52:36.593469 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2730634a5cfd4c2fb5c4e30100c556d4" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
W20250112 02:52:36.593544 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet fe614eaeb793473d9b2da206b7345562 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe614eaeb793473d9b2da206b7345562 already in progress: creating tablet
I20250112 02:52:36.593688 919 raft_consensus.cc:2388] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1e775296eb746db8b318f17d26820e6 in current term 1: Already voted for candidate 289afc116aa74408aa5bff74e11c50d8 in this term.
I20250112 02:52:36.593729 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet a1a0a13696fc41ccb28218f1275a4b5d (table default.Table_62be2ee0d4744f198098621bf776cf01 [id=bf2df21352c64c069c2a06c878911016]) successfully deleted
I20250112 02:52:36.593880 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet b2d8ea2ef18a46508371b4789ab6f4fa (table default.Table_12a25f7a620047b69f34be9eeae64dd2 [id=87c1bff49b4346599bf98166ce8672d4]) successfully deleted
I20250112 02:52:36.594117 991 leader_election.cc:304] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:36.599285 2024 ts_tablet_manager.cc:1918] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250112 02:52:36.599548 714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 208375us
I20250112 02:52:36.600327 2339 tablet.cc:1721] T 48634d6d78784fae98512b242e5476df P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:36.601488 714 leader_election.cc:304] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:36.601751 1609 raft_consensus.cc:2798] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:36.601794 2024 log.cc:1198] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/35f59617b34046568dd83bbb7cdc42d2
I20250112 02:52:36.601842 1609 raft_consensus.cc:491] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:36.601941 1609 raft_consensus.cc:3054] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.602200 2024 ts_tablet_manager.cc:1939] T 35f59617b34046568dd83bbb7cdc42d2 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:36.602963 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 46) took 2170 ms. Trace:
I20250112 02:52:36.603044 2024 rpcz_store.cc:276] 0112 02:52:34.432224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.432473 (+ 249us) service_pool.cc:225] Handling call
0112 02:52:36.581315 (+2148842us) ts_tablet_manager.cc:1145] Deleting tablet 35f59617b34046568dd83bbb7cdc42d2
0112 02:52:36.581326 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.596376 (+ 15050us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.599224 (+ 2848us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.602945 (+ 3721us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2148479}
I20250112 02:52:36.603331 2024 tablet_replica.cc:331] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:36.603547 2024 raft_consensus.cc:2238] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:36.603843 2024 raft_consensus.cc:2267] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.604174 1609 raft_consensus.cc:513] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.604306 2024 ts_tablet_manager.cc:1905] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:36.605368 713 leader_election.cc:343] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: ac716c6da01941a989754aac1ef430d5
I20250112 02:52:36.605563 1609 leader_election.cc:290] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:36.606019 2215 raft_consensus.cc:2798] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:36.606194 2215 raft_consensus.cc:695] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:36.570257 2003 rpcz_store.cc:276] 0112 02:52:34.188883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.188940 (+ 57us) service_pool.cc:225] Handling call
0112 02:52:36.505918 (+2316978us) ts_tablet_manager.cc:1145] Deleting tablet 61593969d2234ce3a6089027b067d922
0112 02:52:36.505926 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.532093 (+ 26167us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.569010 (+ 36917us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.570114 (+ 1104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2312354}
I20250112 02:52:36.606709 2003 tablet_replica.cc:331] T 64707ee432fb4e96a3bbf7252dfa1813 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:36.606913 2003 raft_consensus.cc:2238] T 64707ee432fb4e96a3bbf7252dfa1813 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.606490 2215 consensus_queue.cc:237] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.607055 2003 raft_consensus.cc:2267] T 64707ee432fb4e96a3bbf7252dfa1813 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.608618 1181 tablet_bootstrap.cc:492] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:36.608739 1181 ts_tablet_manager.cc:1397] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.001s
W20250112 02:52:36.609309 714 leader_election.cc:343] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:36.609467 1181 raft_consensus.cc:357] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.609650 1181 raft_consensus.cc:383] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.609699 1181 raft_consensus.cc:738] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.610323 1181 consensus_queue.cc:260] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.610663 2003 ts_tablet_manager.cc:1905] T 64707ee432fb4e96a3bbf7252dfa1813 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.610781 1181 ts_tablet_manager.cc:1428] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:36.611012 1181 tablet_bootstrap.cc:492] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:36.613363 1181 tablet_bootstrap.cc:654] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.614835 989 leader_election.cc:304] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:36.614886 1181 tablet_bootstrap.cc:492] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:36.614993 1181 ts_tablet_manager.cc:1397] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20250112 02:52:36.615056 2207 raft_consensus.cc:3054] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.615489 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 35f59617b34046568dd83bbb7cdc42d2 (table default.Table_cfef2f0eb8dd4927b6d0af0a8c3c739c [id=a0145eb8ddc14728b448e5c08a4d9311]) successfully deleted
I20250112 02:52:36.616012 1181 raft_consensus.cc:357] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.616183 1181 raft_consensus.cc:383] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.616250 1181 raft_consensus.cc:738] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.616417 1181 consensus_queue.cc:260] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.616845 1181 ts_tablet_manager.cc:1428] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:36.617028 1181 tablet_bootstrap.cc:492] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:36.617586 2207 raft_consensus.cc:2743] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:36.618364 1181 tablet_bootstrap.cc:654] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.621766 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d25815a5df94e099e7481c48c3da793" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:36.621992 2024 ts_tablet_manager.cc:1918] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.622078 2024 log.cc:1198] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/42cb7e7a41634cf1bdb978ac9d2befcd
I20250112 02:52:36.622380 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 3c686fdc8896463c81c9f7c5f53a3da2 (table default.Table_4934f6022f2441eda88751236a5798be [id=5c4769bfcefd4fe0a0f3c2e315d50c32]) successfully deleted
I20250112 02:52:36.622658 2024 ts_tablet_manager.cc:1939] T 42cb7e7a41634cf1bdb978ac9d2befcd P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:36.622650 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e933b58dab0b45e8bf1c49eebb1c53e3" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:36.622860 1173 tablet_bootstrap.cc:492] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:36.622975 1173 ts_tablet_manager.cc:1397] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.273s user 0.012s sys 0.005s
I20250112 02:52:36.623206 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 49bb6d20cfa0453e90e048825114525e (table default.Table_f359341b73c14c60b859f06dc5f428a7 [id=bedc9786ee5b40a9bdcbd3cd0be38361]) successfully deleted
I20250112 02:52:36.623476 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e933b58dab0b45e8bf1c49eebb1c53e3" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:36.623996 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 47) took 2191 ms. Trace:
I20250112 02:52:36.624037 1173 raft_consensus.cc:357] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.624173 2024 rpcz_store.cc:276] 0112 02:52:34.432753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.437789 (+ 5036us) service_pool.cc:225] Handling call
0112 02:52:36.603290 (+2165501us) ts_tablet_manager.cc:1145] Deleting tablet 42cb7e7a41634cf1bdb978ac9d2befcd
0112 02:52:36.603300 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.607254 (+ 3954us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.621911 (+ 14657us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.623955 (+ 2044us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2162894}
I20250112 02:52:36.624743 2024 tablet_replica.cc:331] T 7f4e4f1e451b48dc9cdd780300f6888c P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:36.625156 2024 raft_consensus.cc:2238] T 7f4e4f1e451b48dc9cdd780300f6888c P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.625267 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac716c6da01941a989754aac1ef430d5" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:36.625557 1051 raft_consensus.cc:3054] T ac716c6da01941a989754aac1ef430d5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.625859 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 61593969d2234ce3a6089027b067d922 (table default.Table_0188036e1bd0455ea82fd355934295fc [id=b2f29883bc3844cf8b008345b849345c]) successfully deleted
I20250112 02:52:36.625909 1609 raft_consensus.cc:491] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.626053 1609 raft_consensus.cc:513] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.626542 1609 leader_election.cc:290] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:36.627113 714 leader_election.cc:343] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:36.627252 713 leader_election.cc:343] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 352c87f1dc724e7b94bc2e64abfce8d7
I20250112 02:52:36.628010 1051 raft_consensus.cc:2463] T ac716c6da01941a989754aac1ef430d5 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:36.527436 2036 log.cc:1198] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/015412a5fa304453bd8d1be1c3cd7a46
I20250112 02:52:36.628553 2036 ts_tablet_manager.cc:1939] T 015412a5fa304453bd8d1be1c3cd7a46 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:36.628736 2024 raft_consensus.cc:2267] T 7f4e4f1e451b48dc9cdd780300f6888c P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.629284 2024 ts_tablet_manager.cc:1905] T 7f4e4f1e451b48dc9cdd780300f6888c P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.629269 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 56) took 2389 ms. Trace:
I20250112 02:52:36.629341 2036 rpcz_store.cc:276] 0112 02:52:34.239329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.277383 (+ 38054us) service_pool.cc:225] Handling call
0112 02:52:36.431258 (+2153875us) ts_tablet_manager.cc:1145] Deleting tablet 015412a5fa304453bd8d1be1c3cd7a46
0112 02:52:36.431269 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.524586 (+ 93317us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.527270 (+ 2684us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.629251 (+101981us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2153755}
I20250112 02:52:36.629820 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 015412a5fa304453bd8d1be1c3cd7a46 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]) successfully deleted
I20250112 02:52:36.630131 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 58) took 2389 ms. Trace:
I20250112 02:52:36.630286 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 42cb7e7a41634cf1bdb978ac9d2befcd (table default.Table_3530ac60f1e04607aafce2de389968df [id=8245f0e407da4a88ad69764267feed0e]) successfully deleted
I20250112 02:52:36.630267 2036 rpcz_store.cc:276] 0112 02:52:34.240963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.277511 (+ 36548us) service_pool.cc:225] Handling call
0112 02:52:36.630041 (+2352530us) ts_tablet_manager.cc:1145] Deleting tablet d63d411f906b4d4eb68343bdcc493e05
0112 02:52:36.630049 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.630117 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2352421}
W20250112 02:52:36.630743 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet d63d411f906b4d4eb68343bdcc493e05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d63d411f906b4d4eb68343bdcc493e05 already in progress: creating tablet
I20250112 02:52:36.631013 2036 tablet_replica.cc:331] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:36.631237 2036 raft_consensus.cc:2238] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.631402 2036 raft_consensus.cc:2267] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.632006 2036 ts_tablet_manager.cc:1905] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.635380 1181 tablet_bootstrap.cc:492] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:36.635512 1181 ts_tablet_manager.cc:1397] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.002s
I20250112 02:52:36.636237 1181 raft_consensus.cc:357] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.636402 1181 raft_consensus.cc:383] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.636600 1181 raft_consensus.cc:738] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.636798 1181 consensus_queue.cc:260] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.637310 1181 ts_tablet_manager.cc:1428] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250112 02:52:36.638845 714 leader_election.cc:343] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:36.639150 714 leader_election.cc:343] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:36.624258 1173 raft_consensus.cc:383] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.639456 1173 raft_consensus.cc:738] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.639521 714 leader_election.cc:304] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:36.639784 2024 ts_tablet_manager.cc:1918] T 7f4e4f1e451b48dc9cdd780300f6888c P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.639760 1173 consensus_queue.cc:260] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.640048 2024 log.cc:1198] T 7f4e4f1e451b48dc9cdd780300f6888c P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/7f4e4f1e451b48dc9cdd780300f6888c
I20250112 02:52:36.640357 1173 ts_tablet_manager.cc:1428] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250112 02:52:36.640519 2024 ts_tablet_manager.cc:1939] T 7f4e4f1e451b48dc9cdd780300f6888c P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:36.640619 1173 tablet_bootstrap.cc:492] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:36.641300 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 48) took 2203 ms. Trace:
I20250112 02:52:36.641892 2024 rpcz_store.cc:276] 0112 02:52:34.438098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.438659 (+ 561us) service_pool.cc:225] Handling call
0112 02:52:36.624680 (+2186021us) ts_tablet_manager.cc:1145] Deleting tablet 7f4e4f1e451b48dc9cdd780300f6888c
0112 02:52:36.624690 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.634002 (+ 9312us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.639371 (+ 5369us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.641230 (+ 1859us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2184180}
I20250112 02:52:36.642339 2024 tablet_replica.cc:331] T ac716c6da01941a989754aac1ef430d5 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:36.642570 2024 raft_consensus.cc:2238] T ac716c6da01941a989754aac1ef430d5 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.642729 1173 tablet_bootstrap.cc:654] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.642822 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 7f4e4f1e451b48dc9cdd780300f6888c (table default.Table_096703f878244b8dac2dcf187fd23fd6 [id=e1779f88b20942eaa2c640ac24946fff]) successfully deleted
I20250112 02:52:36.643141 2024 raft_consensus.cc:2267] T ac716c6da01941a989754aac1ef430d5 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.644093 2024 ts_tablet_manager.cc:1905] T ac716c6da01941a989754aac1ef430d5 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.644651 2190 raft_consensus.cc:2798] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:36.644815 2190 raft_consensus.cc:695] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:36.645109 2190 consensus_queue.cc:237] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.647372 1181 tablet_bootstrap.cc:492] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:36.647883 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352c87f1dc724e7b94bc2e64abfce8d7" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:36.648083 1051 raft_consensus.cc:2463] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:36.649233 1181 tablet_bootstrap.cc:654] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.651260 714 leader_election.cc:304] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:36.651552 2179 raft_consensus.cc:2798] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:36.651639 2179 raft_consensus.cc:491] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:36.651728 2179 raft_consensus.cc:3054] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.654095 2179 raft_consensus.cc:513] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.654654 2179 leader_election.cc:290] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:36.655196 713 leader_election.cc:343] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 352c87f1dc724e7b94bc2e64abfce8d7
I20250112 02:52:36.655436 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352c87f1dc724e7b94bc2e64abfce8d7" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:36.655663 1051 raft_consensus.cc:3054] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.658725 1051 raft_consensus.cc:2463] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:36.660243 1181 tablet_bootstrap.cc:492] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:36.660346 1181 ts_tablet_manager.cc:1397] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20250112 02:52:36.660878 1181 raft_consensus.cc:357] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.661011 1181 raft_consensus.cc:383] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.661048 1181 raft_consensus.cc:738] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.661235 1181 consensus_queue.cc:260] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.661650 1181 ts_tablet_manager.cc:1428] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:36.662320 1181 tablet_bootstrap.cc:492] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:36.663336 714 leader_election.cc:304] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:36.663673 2190 raft_consensus.cc:2798] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:36.663761 2003 ts_tablet_manager.cc:1918] T 64707ee432fb4e96a3bbf7252dfa1813 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.663900 2003 log.cc:1198] T 64707ee432fb4e96a3bbf7252dfa1813 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/64707ee432fb4e96a3bbf7252dfa1813
I20250112 02:52:36.663990 1181 tablet_bootstrap.cc:654] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.664232 2003 ts_tablet_manager.cc:1939] T 64707ee432fb4e96a3bbf7252dfa1813 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:36.664753 2190 raft_consensus.cc:695] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:36.665066 2190 consensus_queue.cc:237] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:36.665993 854 consensus_peers.cc:487] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f4e4f1e451b48dc9cdd780300f6888c. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:36.667023 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_2312878f87074a63af9c96478ffb7416"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:36.667645 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2312878f87074a63af9c96478ffb7416 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:36.675431 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 50) took 2482 ms. Trace:
I20250112 02:52:36.675557 2003 rpcz_store.cc:276] 0112 02:52:34.203612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.203753 (+ 141us) service_pool.cc:225] Handling call
0112 02:52:36.606659 (+2402906us) ts_tablet_manager.cc:1145] Deleting tablet 64707ee432fb4e96a3bbf7252dfa1813
0112 02:52:36.606667 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.634727 (+ 28060us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.637851 (+ 3124us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.675375 (+ 37524us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2393073}
I20250112 02:52:36.675673 2117 raft_consensus.cc:491] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.675770 2117 raft_consensus.cc:513] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.676033 2003 tablet_replica.cc:331] T f7cd0927c6a4401cbe69f55d4db9963a P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:36.676246 2117 leader_election.cc:290] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:36.676247 2003 raft_consensus.cc:2238] T f7cd0927c6a4401cbe69f55d4db9963a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.676411 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 64707ee432fb4e96a3bbf7252dfa1813 (table default.Table_c26869f6cbd540fe980dcf207b5c729d [id=cdd6a2097792484d84bca856f30d7ac1]) successfully deleted
I20250112 02:52:36.677448 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc8658b0efe43bdb339edbff64973f6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:36.677873 854 leader_election.cc:343] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:36.678995 2117 raft_consensus.cc:491] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1e775296eb746db8b318f17d26820e6)
I20250112 02:52:36.679112 2117 raft_consensus.cc:513] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:36.679620 2117 leader_election.cc:290] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:36.676934 2003 raft_consensus.cc:2267] T f7cd0927c6a4401cbe69f55d4db9963a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.681711 2003 ts_tablet_manager.cc:1905] T f7cd0927c6a4401cbe69f55d4db9963a P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.681903 452 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 134) took 1706 ms. Trace:
I20250112 02:52:36.682013 452 rpcz_store.cc:276] 0112 02:52:34.975684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:36.572945 (+1597261us) service_pool.cc:225] Handling call
0112 02:52:36.666918 (+ 93973us) spinlock_profiling.cc:243] Waited 93.7 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd72d03 00007f3e13e75236 00007f3e13e74f65 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:36.667736 (+ 818us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:36.668183 (+ 447us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:36.681611 (+ 13428us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:36.681736 (+ 125us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:36.681829 (+ 93us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:36.669263 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:36.669341 (+ 78us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:36.669347 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:36.669351 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:36.669458 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:36.669469 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:36.669475 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:36.669553 (+ 78us) tablet.cc:698] Row locks acquired
0112 02:52:36.669556 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:36.669591 (+ 35us) write_op.cc:270] Start()
0112 02:52:36.669622 (+ 31us) write_op.cc:276] Timestamp: P: 1736650356669550 usec, L: 0
0112 02:52:36.669625 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:36.669790 (+ 165us) log.cc:844] Serialized 433 byte log entry
0112 02:52:36.680906 (+ 11116us) op_driver.cc:464] REPLICATION: finished
0112 02:52:36.680972 (+ 66us) write_op.cc:301] APPLY: starting
0112 02:52:36.680983 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:36.681015 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:36.681047 (+ 32us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:36.681178 (+ 131us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:36.681205 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:36.681214 (+ 9us) write_op.cc:312] APPLY: finished
0112 02:52:36.681269 (+ 55us) log.cc:844] Serialized 29 byte log entry
0112 02:52:36.681318 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.681336 (+ 18us) write_op.cc:454] Released schema lock
0112 02:52:36.681343 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":215568768,"child_traces":[["op",{"apply.queue_time_us":31,"apply.run_cpu_time_us":425,"apply.run_wall_time_us":424,"num_ops":2,"prepare.queue_time_us":612,"prepare.run_cpu_time_us":599,"prepare.run_wall_time_us":598,"raft.queue_time_us":867,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":11258}]]}
I20250112 02:52:36.682637 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d0f67141dc46b6a87ca0ad080d5bfe" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:36.683065 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_8dde1cf3a26244709cf38e8563b97a2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:36.683522 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8dde1cf3a26244709cf38e8563b97a2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:36.677332 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc8658b0efe43bdb339edbff64973f6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:36.684150 855 leader_election.cc:343] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:36.684245 855 leader_election.cc:304] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:36.684827 854 leader_election.cc:343] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 02d0f67141dc46b6a87ca0ad080d5bfe
I20250112 02:52:36.684912 854 leader_election.cc:304] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:36.685242 2222 raft_consensus.cc:2743] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:36.685384 2222 raft_consensus.cc:2743] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:52:36.693267 2003 ts_tablet_manager.cc:1918] T f7cd0927c6a4401cbe69f55d4db9963a P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:36.693389 2003 log.cc:1198] T f7cd0927c6a4401cbe69f55d4db9963a P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/f7cd0927c6a4401cbe69f55d4db9963a
I20250112 02:52:36.693775 2003 ts_tablet_manager.cc:1939] T f7cd0927c6a4401cbe69f55d4db9963a P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:36.694475 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 48) took 2526 ms. Trace:
I20250112 02:52:36.696451 1177 tablet_bootstrap.cc:492] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:36.696584 1177 ts_tablet_manager.cc:1397] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.138s user 0.014s sys 0.007s
I20250112 02:52:36.694586 2003 rpcz_store.cc:276] 0112 02:52:34.168132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.199903 (+ 31771us) service_pool.cc:225] Handling call
0112 02:52:36.675937 (+2476034us) ts_tablet_manager.cc:1145] Deleting tablet f7cd0927c6a4401cbe69f55d4db9963a
0112 02:52:36.675947 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.686274 (+ 10327us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.693151 (+ 6877us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.694431 (+ 1280us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2456456}
I20250112 02:52:36.696931 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet f7cd0927c6a4401cbe69f55d4db9963a (table default.Table_8b4baeb21bec40af9997451f3eceddf1 [id=b7a72bcbde114d5992782721007a0cd4]) successfully deleted
I20250112 02:52:36.696991 2003 tablet_replica.cc:331] T 8a12b225131b464494a9f02133d37aea P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:36.697170 2003 raft_consensus.cc:2238] T 8a12b225131b464494a9f02133d37aea P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.697309 2003 raft_consensus.cc:2267] T 8a12b225131b464494a9f02133d37aea P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.697340 1177 raft_consensus.cc:357] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.697537 1177 raft_consensus.cc:383] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.697625 1177 raft_consensus.cc:738] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.697983 1177 consensus_queue.cc:260] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.698977 1177 ts_tablet_manager.cc:1428] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:36.699229 1177 tablet_bootstrap.cc:492] T d4abf59d96414e5faa7cf75ba133832c P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:36.701072 2117 raft_consensus.cc:491] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.701228 2117 raft_consensus.cc:513] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.701773 2117 leader_election.cc:290] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:36.701936 452 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 135) took 1715 ms. Trace:
I20250112 02:52:36.702024 452 rpcz_store.cc:276] 0112 02:52:34.986631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:36.682593 (+1695962us) service_pool.cc:225] Handling call
0112 02:52:36.683633 (+ 1040us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:36.684029 (+ 396us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:36.701707 (+ 17678us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:36.701810 (+ 103us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:36.701885 (+ 75us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:36.684458 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:36.684544 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:36.684548 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:36.684550 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:36.684615 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:36.684623 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:36.684627 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:36.684679 (+ 52us) tablet.cc:698] Row locks acquired
0112 02:52:36.684681 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:36.684705 (+ 24us) write_op.cc:270] Start()
0112 02:52:36.684728 (+ 23us) write_op.cc:276] Timestamp: P: 1736650356684668 usec, L: 0
0112 02:52:36.684733 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:36.684855 (+ 122us) log.cc:844] Serialized 433 byte log entry
0112 02:52:36.700863 (+ 16008us) op_driver.cc:464] REPLICATION: finished
0112 02:52:36.700988 (+ 125us) write_op.cc:301] APPLY: starting
0112 02:52:36.700999 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:36.701034 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:36.701118 (+ 84us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:36.701297 (+ 179us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:36.701328 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:36.701336 (+ 8us) write_op.cc:312] APPLY: finished
0112 02:52:36.701427 (+ 91us) log.cc:844] Serialized 29 byte log entry
0112 02:52:36.701623 (+ 196us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.701636 (+ 13us) write_op.cc:454] Released schema lock
0112 02:52:36.701642 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":587648,"child_traces":[["op",{"apply.queue_time_us":69,"num_ops":2,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":480,"prepare.run_wall_time_us":480,"raft.queue_time_us":53,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":218,"replication_time_us":16108}]]}
I20250112 02:52:36.702780 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_8293882503214cf284e12ff8cc75b9c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:36.703290 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8293882503214cf284e12ff8cc75b9c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:36.704838 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7394b1878447a189ce45957f10e4df" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:36.705216 854 leader_election.cc:343] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:36.701287 2003 ts_tablet_manager.cc:1905] T 8a12b225131b464494a9f02133d37aea P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.702481 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7394b1878447a189ce45957f10e4df" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:36.706152 855 leader_election.cc:343] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:36.706267 855 leader_election.cc:304] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:36.706696 2117 raft_consensus.cc:2743] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:36.708108 756 tablet_service.cc:1467] Processing CreateTablet for tablet b455a37448a044679bd2c0646de3d804 (DEFAULT_TABLE table=default.Table_5981a14692a2470290628d9e5934c4e5 [id=55eb92db6e924911b44fb44221d453e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.708431 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b455a37448a044679bd2c0646de3d804. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.708444 1034 tablet_service.cc:1467] Processing CreateTablet for tablet b455a37448a044679bd2c0646de3d804 (DEFAULT_TABLE table=default.Table_5981a14692a2470290628d9e5934c4e5 [id=55eb92db6e924911b44fb44221d453e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.708761 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b455a37448a044679bd2c0646de3d804. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.708912 898 tablet_service.cc:1467] Processing CreateTablet for tablet b455a37448a044679bd2c0646de3d804 (DEFAULT_TABLE table=default.Table_5981a14692a2470290628d9e5934c4e5 [id=55eb92db6e924911b44fb44221d453e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.710798 899 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8ad3cda8124845824024cb69ff54ba (DEFAULT_TABLE table=default.Table_b30bfb8db92545468698dde44b7c4200 [id=be8a3cbbaed24344a278cbb5e0a5c883]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.712044 742 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8ad3cda8124845824024cb69ff54ba (DEFAULT_TABLE table=default.Table_b30bfb8db92545468698dde44b7c4200 [id=be8a3cbbaed24344a278cbb5e0a5c883]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.712218 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8ad3cda8124845824024cb69ff54ba (DEFAULT_TABLE table=default.Table_b30bfb8db92545468698dde44b7c4200 [id=be8a3cbbaed24344a278cbb5e0a5c883]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.712406 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a8ad3cda8124845824024cb69ff54ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.712572 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a8ad3cda8124845824024cb69ff54ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.713829 2117 raft_consensus.cc:491] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.713917 2117 raft_consensus.cc:513] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:36.714336 2117 leader_election.cc:290] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:36.714754 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f924459d1150472aafb325a11825db23" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:36.715394 855 leader_election.cc:343] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:36.715566 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f924459d1150472aafb325a11825db23" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:36.716001 854 leader_election.cc:343] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:36.716084 854 leader_election.cc:304] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:36.716374 2117 raft_consensus.cc:2743] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:36.721756 2003 ts_tablet_manager.cc:1918] T 8a12b225131b464494a9f02133d37aea P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.722000 2003 log.cc:1198] T 8a12b225131b464494a9f02133d37aea P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/8a12b225131b464494a9f02133d37aea
I20250112 02:52:36.724992 1181 tablet_bootstrap.cc:492] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:36.725009 1173 tablet_bootstrap.cc:492] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:36.725145 1173 ts_tablet_manager.cc:1397] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.085s user 0.006s sys 0.017s
I20250112 02:52:36.725165 1181 ts_tablet_manager.cc:1397] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.063s user 0.000s sys 0.011s
I20250112 02:52:36.725854 1181 raft_consensus.cc:357] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.725894 1173 raft_consensus.cc:357] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.726075 1173 raft_consensus.cc:383] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.726146 1173 raft_consensus.cc:738] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.726094 1181 raft_consensus.cc:383] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.726588 1181 raft_consensus.cc:738] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.726884 1181 consensus_queue.cc:260] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.727809 1181 ts_tablet_manager.cc:1428] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 02:52:36.728500 1181 tablet_bootstrap.cc:492] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:36.729449 452 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 136) took 1714 ms. Trace:
I20250112 02:52:36.729544 452 rpcz_store.cc:276] 0112 02:52:35.015440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:36.702568 (+1687128us) service_pool.cc:225] Handling call
0112 02:52:36.703377 (+ 809us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:36.703747 (+ 370us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:36.721292 (+ 17545us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:36.729329 (+ 8037us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:36.729402 (+ 73us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:36.704108 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:36.704194 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:36.704198 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:36.704200 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:36.704259 (+ 59us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:36.704267 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:36.704270 (+ 3us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:36.704349 (+ 79us) tablet.cc:698] Row locks acquired
0112 02:52:36.704351 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:36.704380 (+ 29us) write_op.cc:270] Start()
0112 02:52:36.704405 (+ 25us) write_op.cc:276] Timestamp: P: 1736650356704338 usec, L: 0
0112 02:52:36.704409 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:36.704546 (+ 137us) log.cc:844] Serialized 433 byte log entry
0112 02:52:36.718425 (+ 13879us) op_driver.cc:464] REPLICATION: finished
0112 02:52:36.718532 (+ 107us) write_op.cc:301] APPLY: starting
0112 02:52:36.718545 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:36.718583 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:36.718638 (+ 55us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:36.718739 (+ 101us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:36.718771 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:36.718779 (+ 8us) write_op.cc:312] APPLY: finished
0112 02:52:36.718839 (+ 60us) log.cc:844] Serialized 29 byte log entry
0112 02:52:36.718889 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.718907 (+ 18us) write_op.cc:454] Released schema lock
0112 02:52:36.718914 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:36.718976 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.718982 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":66,"apply.run_cpu_time_us":435,"apply.run_wall_time_us":435,"num_ops":2,"prepare.queue_time_us":31,"prepare.run_cpu_time_us":512,"prepare.run_wall_time_us":512,"raft.queue_time_us":9127,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":7679,"replication_time_us":13987}]]}
I20250112 02:52:36.730218 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_a39e287f411145fca081afaaf41d3311"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:36.730636 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a39e287f411145fca081afaaf41d3311 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:36.731235 1181 tablet_bootstrap.cc:654] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.733203 1173 consensus_queue.cc:260] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.733785 1173 ts_tablet_manager.cc:1428] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250112 02:52:36.734761 1173 tablet_bootstrap.cc:492] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:36.737490 2024 ts_tablet_manager.cc:1918] T ac716c6da01941a989754aac1ef430d5 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:36.737599 2024 log.cc:1198] T ac716c6da01941a989754aac1ef430d5 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/ac716c6da01941a989754aac1ef430d5
I20250112 02:52:36.737897 2024 ts_tablet_manager.cc:1939] T ac716c6da01941a989754aac1ef430d5 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:36.738370 1177 tablet_bootstrap.cc:654] T d4abf59d96414e5faa7cf75ba133832c P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.739029 1181 tablet_bootstrap.cc:492] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
W20250112 02:52:36.739692 439 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:38771, user_credentials={real_user=slave}} blocked reactor thread for 66743.4us
I20250112 02:52:36.739753 1181 ts_tablet_manager.cc:1397] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20250112 02:52:36.740861 1181 raft_consensus.cc:357] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.741307 1173 tablet_bootstrap.cc:654] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.743148 1181 raft_consensus.cc:383] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.742712 2222 raft_consensus.cc:491] T 781fe3514f4c4f80b1ee49278502ccc1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.743389 1181 raft_consensus.cc:738] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.743534 2117 raft_consensus.cc:491] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.743860 1181 consensus_queue.cc:260] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.744045 2117 raft_consensus.cc:513] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:36.744530 2117 leader_election.cc:290] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:36.745321 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416fb6820cd54e118950c68d381f51b2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:36.743484 2222 raft_consensus.cc:513] T 781fe3514f4c4f80b1ee49278502ccc1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.745952 1181 ts_tablet_manager.cc:1428] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250112 02:52:36.746078 2222 leader_election.cc:290] T 781fe3514f4c4f80b1ee49278502ccc1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:36.742130 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b455a37448a044679bd2c0646de3d804. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.746158 1181 tablet_bootstrap.cc:492] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:36.746317 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a8ad3cda8124845824024cb69ff54ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.750085 1181 tablet_bootstrap.cc:654] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:36.750281 855 leader_election.cc:343] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:36.750829 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416fb6820cd54e118950c68d381f51b2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:36.750923 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "781fe3514f4c4f80b1ee49278502ccc1" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:36.751240 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "781fe3514f4c4f80b1ee49278502ccc1" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:36.752305 1173 tablet_bootstrap.cc:492] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:36.752444 1173 ts_tablet_manager.cc:1397] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.001s
W20250112 02:52:36.752552 855 leader_election.cc:343] T 781fe3514f4c4f80b1ee49278502ccc1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:36.753049 854 leader_election.cc:343] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:36.753031 1173 raft_consensus.cc:357] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.753144 854 leader_election.cc:304] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:36.753201 1173 raft_consensus.cc:383] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.753263 1173 raft_consensus.cc:738] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
W20250112 02:52:36.753487 854 leader_election.cc:343] T 781fe3514f4c4f80b1ee49278502ccc1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:36.753495 1173 consensus_queue.cc:260] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.753553 854 leader_election.cc:304] T 781fe3514f4c4f80b1ee49278502ccc1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:36.753893 1173 ts_tablet_manager.cc:1428] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 02:52:36.753974 2036 ts_tablet_manager.cc:1918] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:36.754066 2036 log.cc:1198] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/f05c8426ea3d4fe89fbe6c88c688019f
I20250112 02:52:36.754227 2222 raft_consensus.cc:2743] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:36.754333 2036 ts_tablet_manager.cc:1939] T f05c8426ea3d4fe89fbe6c88c688019f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:36.754357 2222 raft_consensus.cc:2743] T 781fe3514f4c4f80b1ee49278502ccc1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:36.755756 1173 tablet_bootstrap.cc:492] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:36.756588 1181 tablet_bootstrap.cc:492] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:36.756693 1181 ts_tablet_manager.cc:1397] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250112 02:52:36.757246 1173 tablet_bootstrap.cc:654] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.757344 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:36.757308 1181 raft_consensus.cc:357] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.757457 1181 raft_consensus.cc:383] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.757529 1181 raft_consensus.cc:738] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.757673 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:36.758041 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:36.758370 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:36.758517 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
I20250112 02:52:36.758677 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet f3771166b77e4f90aaa116ab1a9dc299 (table default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6]). Expected version 1 got 0
I20250112 02:52:36.758824 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:36.759317 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet fccff64af8ca4400afc116db98918119 (table default.Table_15bcb2c0f2a74458a1bed231fd3437fc [id=ca9c8281bf254f248076cb600e7c4f87]). Expected version 2 got 0
W20250112 02:52:36.760809 439 consensus_peers.cc:487] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e -> Peer f4a49a5756b143fcb767c3f41463f8d9 (127.0.104.61:38771): Couldn't send request to peer f4a49a5756b143fcb767c3f41463f8d9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38771: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250112 02:52:36.761880 1181 consensus_queue.cc:260] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.762269 452 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 137) took 1664 ms. Trace:
I20250112 02:52:36.762367 452 rpcz_store.cc:276] 0112 02:52:35.097525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:36.730003 (+1632478us) service_pool.cc:225] Handling call
0112 02:52:36.730732 (+ 729us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:36.731078 (+ 346us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:36.762035 (+ 30957us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:36.762135 (+ 100us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:36.762204 (+ 69us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:36.732193 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:36.732284 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:36.732289 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:36.732304 (+ 15us) tablet.cc:659] Decoding operations
0112 02:52:36.732368 (+ 64us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:36.732379 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:36.732383 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:36.732436 (+ 53us) tablet.cc:698] Row locks acquired
0112 02:52:36.732438 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:36.732468 (+ 30us) write_op.cc:270] Start()
0112 02:52:36.732489 (+ 21us) write_op.cc:276] Timestamp: P: 1736650356732422 usec, L: 0
0112 02:52:36.732492 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:36.732624 (+ 132us) log.cc:844] Serialized 433 byte log entry
0112 02:52:36.738492 (+ 5868us) op_driver.cc:464] REPLICATION: finished
0112 02:52:36.761475 (+ 22983us) write_op.cc:301] APPLY: starting
0112 02:52:36.761491 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:36.761528 (+ 37us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:36.761550 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:36.761644 (+ 94us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:36.761682 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:36.761691 (+ 9us) write_op.cc:312] APPLY: finished
0112 02:52:36.761757 (+ 66us) log.cc:844] Serialized 29 byte log entry
0112 02:52:36.761946 (+ 189us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.761964 (+ 18us) write_op.cc:454] Released schema lock
0112 02:52:36.761970 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":22919,"num_ops":2,"prepare.queue_time_us":693,"prepare.run_cpu_time_us":505,"prepare.run_wall_time_us":505,"raft.queue_time_us":3620,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":24226,"replication_time_us":5974}]]}
I20250112 02:52:36.763013 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_93133ac10ebe44d58e69c21d228c1c5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:36.763458 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93133ac10ebe44d58e69c21d228c1c5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:36.764698 826 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:36.767704 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 49) took 2328 ms. Trace:
I20250112 02:52:36.767823 2024 rpcz_store.cc:276] 0112 02:52:34.438922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.439863 (+ 941us) service_pool.cc:225] Handling call
0112 02:52:36.642257 (+2202394us) ts_tablet_manager.cc:1145] Deleting tablet ac716c6da01941a989754aac1ef430d5
0112 02:52:36.642266 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.695081 (+ 52815us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.737393 (+ 42312us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.767670 (+ 30277us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2201322}
I20250112 02:52:36.768257 2024 tablet_replica.cc:331] T cc62dcbf67e84e36ae1fbe127ccd4879 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:36.768509 2024 raft_consensus.cc:2238] T cc62dcbf67e84e36ae1fbe127ccd4879 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.768715 2024 raft_consensus.cc:2267] T cc62dcbf67e84e36ae1fbe127ccd4879 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.769289 2024 ts_tablet_manager.cc:1905] T cc62dcbf67e84e36ae1fbe127ccd4879 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.769659 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet ac716c6da01941a989754aac1ef430d5 (table default.Table_bee5a719ea4b44b78254bae566a0ba92 [id=d3fa1ea0552e4aaeabd6ac3ac84ca389]) successfully deleted
I20250112 02:52:36.770664 2003 ts_tablet_manager.cc:1939] T 8a12b225131b464494a9f02133d37aea P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:36.770963 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 51) took 2553 ms. Trace:
I20250112 02:52:36.771389 452 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 138) took 1629 ms. Trace:
I20250112 02:52:36.771556 452 rpcz_store.cc:276] 0112 02:52:35.141572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:36.762824 (+1621252us) service_pool.cc:225] Handling call
0112 02:52:36.763821 (+ 997us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:36.764248 (+ 427us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:36.771123 (+ 6875us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.771128 (+ 5us) write_op.cc:454] Released schema lock
0112 02:52:36.771183 (+ 55us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:36.771277 (+ 94us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:36.771340 (+ 63us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:36.764751 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:36.764875 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:36.764881 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:36.764883 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:36.765234 (+ 351us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:36.765245 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:36.765250 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:36.765325 (+ 75us) tablet.cc:698] Row locks acquired
0112 02:52:36.765327 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:36.765353 (+ 26us) write_op.cc:270] Start()
0112 02:52:36.765378 (+ 25us) write_op.cc:276] Timestamp: P: 1736650356765299 usec, L: 0
0112 02:52:36.765381 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:36.765519 (+ 138us) log.cc:844] Serialized 433 byte log entry
0112 02:52:36.768481 (+ 2962us) op_driver.cc:464] REPLICATION: finished
0112 02:52:36.768609 (+ 128us) write_op.cc:301] APPLY: starting
0112 02:52:36.768621 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:36.768650 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:36.768684 (+ 34us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:36.768779 (+ 95us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:36.768809 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:36.768815 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:36.768868 (+ 53us) log.cc:844] Serialized 29 byte log entry
0112 02:52:36.768914 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.768928 (+ 14us) write_op.cc:454] Released schema lock
0112 02:52:36.768933 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":86,"apply.run_cpu_time_us":384,"apply.run_wall_time_us":383,"num_ops":2,"prepare.queue_time_us":84,"prepare.run_cpu_time_us":854,"prepare.run_wall_time_us":854,"raft.queue_time_us":574,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":192,"replication_time_us":3056}]]}
I20250112 02:52:36.772840 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_ac83a75456fe4f0ea7f10cb28e29cd28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:36.773379 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac83a75456fe4f0ea7f10cb28e29cd28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:36.774528 1173 tablet_bootstrap.cc:492] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:36.774664 1173 ts_tablet_manager.cc:1397] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.003s
I20250112 02:52:36.775265 1173 raft_consensus.cc:357] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.775455 1173 raft_consensus.cc:383] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.775554 1173 raft_consensus.cc:738] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.775740 1173 consensus_queue.cc:260] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.776894 1173 ts_tablet_manager.cc:1428] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:36.777192 1173 tablet_bootstrap.cc:492] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:36.777386 2368 raft_consensus.cc:491] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.777477 2368 raft_consensus.cc:513] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.777686 1177 tablet_bootstrap.cc:492] T d4abf59d96414e5faa7cf75ba133832c P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:36.777782 1177 ts_tablet_manager.cc:1397] T d4abf59d96414e5faa7cf75ba133832c P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.079s user 0.003s sys 0.000s
I20250112 02:52:36.778321 2368 leader_election.cc:290] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:36.778694 1181 ts_tablet_manager.cc:1428] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20250112 02:52:36.778693 1177 raft_consensus.cc:357] T d4abf59d96414e5faa7cf75ba133832c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.778856 1177 raft_consensus.cc:383] T d4abf59d96414e5faa7cf75ba133832c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.778892 1173 tablet_bootstrap.cc:654] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.778903 1177 raft_consensus.cc:738] T d4abf59d96414e5faa7cf75ba133832c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.778923 1181 tablet_bootstrap.cc:492] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:36.779057 1177 consensus_queue.cc:260] T d4abf59d96414e5faa7cf75ba133832c P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.779546 1177 ts_tablet_manager.cc:1428] T d4abf59d96414e5faa7cf75ba133832c P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:36.771034 2003 rpcz_store.cc:276] 0112 02:52:34.217204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.217305 (+ 101us) service_pool.cc:225] Handling call
0112 02:52:36.696931 (+2479626us) ts_tablet_manager.cc:1145] Deleting tablet 8a12b225131b464494a9f02133d37aea
0112 02:52:36.696940 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.716790 (+ 19850us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.721662 (+ 4872us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.770950 (+ 49288us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2454593}
I20250112 02:52:36.779863 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 8a12b225131b464494a9f02133d37aea (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]) successfully deleted
I20250112 02:52:36.780018 1177 tablet_bootstrap.cc:492] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:36.780210 2377 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:38771 every 8 connection(s)
I20250112 02:52:36.780578 2003 tablet_replica.cc:331] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:36.780716 2003 raft_consensus.cc:2238] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:36.780689 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4348fe4c32704bf7b3a8d21ff3b8ac1b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:36.781080 855 leader_election.cc:343] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:36.781275 1177 tablet_bootstrap.cc:654] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.781301 2003 raft_consensus.cc:2267] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.781765 2003 ts_tablet_manager.cc:1905] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.781950 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4348fe4c32704bf7b3a8d21ff3b8ac1b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:36.782380 854 leader_election.cc:343] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:36.782483 854 leader_election.cc:304] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:36.786742 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet f924459d1150472aafb325a11825db23 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]). Expected version 3 got 0
I20250112 02:52:36.786929 1778 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:38771
I20250112 02:52:36.787992 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:36.789189 1778 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-1/data/info.pb
I20250112 02:52:36.789911 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3f5b418c17084420b867498fda79fb0a (table default.Table_96d1a09a902340ff8e05efb75faf3854 [id=ad0be37e6b6f4b9597dcb4cbd50d5b12]). Expected version 3 got 2
I20250112 02:52:36.793522 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu as pid 1778
I20250112 02:52:36.801676 1173 tablet_bootstrap.cc:492] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:36.801801 1173 ts_tablet_manager.cc:1397] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.003s
I20250112 02:52:36.802436 1173 raft_consensus.cc:357] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.802655 1173 raft_consensus.cc:383] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.802719 1173 raft_consensus.cc:738] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.802907 1173 consensus_queue.cc:260] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.803375 1173 ts_tablet_manager.cc:1428] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 02:52:36.803700 1173 tablet_bootstrap.cc:492] T b9537fcd96eb41cc8f680075ec71137b P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:36.805402 1173 tablet_bootstrap.cc:654] T b9537fcd96eb41cc8f680075ec71137b P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.806484 2003 ts_tablet_manager.cc:1918] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.806592 2003 log.cc:1198] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/3fd498ccce484a89855c982762086890
I20250112 02:52:36.806932 2003 ts_tablet_manager.cc:1939] T 3fd498ccce484a89855c982762086890 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:36.807762 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 56) took 2565 ms. Trace:
I20250112 02:52:36.807870 2003 rpcz_store.cc:276] 0112 02:52:34.242627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.242675 (+ 48us) service_pool.cc:225] Handling call
0112 02:52:36.780522 (+2537847us) ts_tablet_manager.cc:1145] Deleting tablet 3fd498ccce484a89855c982762086890
0112 02:52:36.780528 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.796287 (+ 15759us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.806212 (+ 9925us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.807738 (+ 1526us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2537634}
I20250112 02:52:36.808264 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 3fd498ccce484a89855c982762086890 (table default.Table_945067c243f84394957a328d65592441 [id=5a5f3290dab54cf0b13c88efd52c90c5]) successfully deleted
I20250112 02:52:36.803597 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:36.808601 2003 tablet_replica.cc:331] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:36.808818 2003 raft_consensus.cc:2238] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:36.809070 2003 raft_consensus.cc:2267] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.809165 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:36.809522 2003 ts_tablet_manager.cc:1905] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.810104 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_5e7392cb5a68412f97bddb2677986918 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 1 got 0
I20250112 02:52:36.811446 2024 ts_tablet_manager.cc:1918] T cc62dcbf67e84e36ae1fbe127ccd4879 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.816001 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:36.816540 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 6117dc8b32a64535a2f4167a03bed0d2 (table default.Table_365739770a11434dbf260ea912e7ea81 [id=868adb627654450c936ef198e49d853b]). Expected version 1 got 0
I20250112 02:52:36.816902 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_5d473cabff2e46e0b531fbce28d9d558 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 1 got 0
I20250112 02:52:36.817137 2024 log.cc:1198] T cc62dcbf67e84e36ae1fbe127ccd4879 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/cc62dcbf67e84e36ae1fbe127ccd4879
I20250112 02:52:36.817533 2024 ts_tablet_manager.cc:1939] T cc62dcbf67e84e36ae1fbe127ccd4879 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:36.817673 2190 consensus_queue.cc:1035] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:36.818097 2222 raft_consensus.cc:2743] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:36.819607 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 59) took 2578 ms. Trace:
I20250112 02:52:36.820053 452 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 139) took 1640 ms. Trace:
I20250112 02:52:36.820437 2003 ts_tablet_manager.cc:1918] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.820559 2003 log.cc:1198] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/015412a5fa304453bd8d1be1c3cd7a46
I20250112 02:52:36.819995 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:36.820132 452 rpcz_store.cc:276] 0112 02:52:35.179609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:36.772620 (+1593011us) service_pool.cc:225] Handling call
0112 02:52:36.773540 (+ 920us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:36.795852 (+ 22312us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x606000052eb8. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 000000000096ceda 00007f3e09786065 00007f3e09785d90 00007f3e1cb7872c 00007f3e1cb729af 00007f3e1cb6b976 00007f3e1cd72e5a 00007f3e13e75236 00007f3e13e74f65 00007f3e105d130e 00007f3e105d01ee
0112 02:52:36.796084 (+ 232us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:36.819805 (+ 23721us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:36.819911 (+ 106us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:36.820000 (+ 89us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:36.797184 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:36.801686 (+ 4502us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:36.801697 (+ 11us) write_op.cc:435] Acquired schema lock
0112 02:52:36.801701 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:36.801835 (+ 134us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:36.801853 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:36.801861 (+ 8us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:36.801970 (+ 109us) tablet.cc:698] Row locks acquired
0112 02:52:36.801974 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:36.802019 (+ 45us) write_op.cc:270] Start()
0112 02:52:36.802062 (+ 43us) write_op.cc:276] Timestamp: P: 1736650356801951 usec, L: 0
0112 02:52:36.802068 (+ 6us) op_driver.cc:348] REPLICATION: starting
0112 02:52:36.802330 (+ 262us) log.cc:844] Serialized 433 byte log entry
0112 02:52:36.810363 (+ 8033us) op_driver.cc:464] REPLICATION: finished
0112 02:52:36.812191 (+ 1828us) write_op.cc:301] APPLY: starting
0112 02:52:36.812205 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:36.812241 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:36.812273 (+ 32us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:36.812419 (+ 146us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:36.812454 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:36.812460 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:36.812518 (+ 58us) log.cc:844] Serialized 29 byte log entry
0112 02:52:36.812571 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.812588 (+ 17us) write_op.cc:454] Released schema lock
0112 02:52:36.812594 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:36.812652 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.812655 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":42805504,"child_traces":[["op",{"apply.queue_time_us":1781,"apply.run_cpu_time_us":459,"apply.run_wall_time_us":458,"num_ops":2,"prepare.queue_time_us":98,"prepare.run_cpu_time_us":992,"prepare.run_wall_time_us":5286,"raft.queue_time_us":1350,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":282,"replication_time_us":8267}]]}
I20250112 02:52:36.820889 2003 ts_tablet_manager.cc:1939] T 015412a5fa304453bd8d1be1c3cd7a46 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:36.821171 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:36.821588 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:36.821725 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 57) took 2578 ms. Trace:
I20250112 02:52:36.821837 2003 rpcz_store.cc:276] 0112 02:52:34.243055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.243095 (+ 40us) service_pool.cc:225] Handling call
0112 02:52:36.808261 (+2565166us) ts_tablet_manager.cc:1145] Deleting tablet 015412a5fa304453bd8d1be1c3cd7a46
0112 02:52:36.808564 (+ 303us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.813798 (+ 5234us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.820355 (+ 6557us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.821704 (+ 1349us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2565001}
I20250112 02:52:36.822467 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 50) took 2352 ms. Trace:
I20250112 02:52:36.822261 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 015412a5fa304453bd8d1be1c3cd7a46 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]) successfully deleted
I20250112 02:52:36.823030 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:36.823199 2024 rpcz_store.cc:276] 0112 02:52:34.469929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.469973 (+ 44us) service_pool.cc:225] Handling call
0112 02:52:36.768182 (+2298209us) ts_tablet_manager.cc:1145] Deleting tablet cc62dcbf67e84e36ae1fbe127ccd4879
0112 02:52:36.768190 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.783840 (+ 15650us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.811343 (+ 27503us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.822394 (+ 11051us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2296962}
I20250112 02:52:36.823661 2024 tablet_replica.cc:331] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:36.823727 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet cc62dcbf67e84e36ae1fbe127ccd4879 (table default.Table_fd0169db0f984d3cbc0d8624dda686bc [id=9e09b6670c6e4e57b62f31fd4755c18b]) successfully deleted
I20250112 02:52:36.824070 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:36.824347 2024 raft_consensus.cc:2238] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:36.824935 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 1 got 0
I20250112 02:52:36.825340 2024 raft_consensus.cc:2267] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.826143 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:36.827047 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:36.827301 2024 ts_tablet_manager.cc:1905] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.831923 2003 tablet_replica.cc:331] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:36.832113 2003 raft_consensus.cc:2238] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:36.819756 2036 rpcz_store.cc:276] 0112 02:52:34.241072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.277631 (+ 36559us) service_pool.cc:225] Handling call
0112 02:52:36.630978 (+2353347us) ts_tablet_manager.cc:1145] Deleting tablet f05c8426ea3d4fe89fbe6c88c688019f
0112 02:52:36.630985 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.742297 (+111312us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.753894 (+ 11597us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.819590 (+ 65696us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4569,"tablet-delete.queue_time_us":2353246}
I20250112 02:52:36.832398 2036 tablet_replica.cc:331] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:36.832418 2003 raft_consensus.cc:2267] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.832532 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet f05c8426ea3d4fe89fbe6c88c688019f (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]) successfully deleted
I20250112 02:52:36.832566 2036 raft_consensus.cc:2238] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:36.832823 2036 raft_consensus.cc:2267] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.832873 2003 ts_tablet_manager.cc:1905] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.832851 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:36.833002 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
I20250112 02:52:36.833123 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:36.833315 2036 ts_tablet_manager.cc:1905] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.833415 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5cc414d6420c41839621ab858067548b (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]). Expected version 1 got 0
I20250112 02:52:36.833585 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:36.834021 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:36.834223 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:36.834514 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 48634d6d78784fae98512b242e5476df (table default.Table_7cbe570a89784c18b4a14a8242062071 [id=fee2f9e65e314ce9a1fc1fb6dfcde317]). Expected version 1 got 0
I20250112 02:52:36.834991 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet f8c01b80fc8242d98fdfbf767212144d (table default.Table_57b3861e9f704dc09d466241c1926fc1 [id=9807191b6e5c4d478cd47a54464fb9ce]). Expected version 2 got 1
I20250112 02:52:36.835357 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:36.835714 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet f31b777e7e0f43a28839d8c48c6a6590 (table default.Table_5fb592e41cc4406aa27546e7b41fdb88 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 1 got 0
I20250112 02:52:36.838164 2003 ts_tablet_manager.cc:1918] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.838253 2003 log.cc:1198] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/b7bd96fb381f4e73a0081be3fc799c36
I20250112 02:52:36.838573 2003 ts_tablet_manager.cc:1939] T b7bd96fb381f4e73a0081be3fc799c36 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:36.839190 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 59) took 2595 ms. Trace:
I20250112 02:52:36.839195 2036 ts_tablet_manager.cc:1918] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.839298 2036 log.cc:1198] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/8a12b225131b464494a9f02133d37aea
I20250112 02:52:36.839305 1177 tablet_bootstrap.cc:492] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:36.839434 1177 ts_tablet_manager.cc:1397] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.060s user 0.007s sys 0.000s
I20250112 02:52:36.840075 1177 raft_consensus.cc:357] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:36.840240 1177 raft_consensus.cc:383] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.840296 1177 raft_consensus.cc:738] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.840643 2036 ts_tablet_manager.cc:1939] T 8a12b225131b464494a9f02133d37aea P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:36.840992 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:36.841380 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 60) took 2593 ms. Trace:
I20250112 02:52:36.841481 2036 rpcz_store.cc:276] 0112 02:52:34.247689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.277744 (+ 30055us) service_pool.cc:225] Handling call
0112 02:52:36.832358 (+2554614us) ts_tablet_manager.cc:1145] Deleting tablet 8a12b225131b464494a9f02133d37aea
0112 02:52:36.832368 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.836438 (+ 4070us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.839131 (+ 2693us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.841360 (+ 2229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2554497}
I20250112 02:52:36.841786 2036 tablet_replica.cc:331] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:36.842003 2036 raft_consensus.cc:2238] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.842133 2036 raft_consensus.cc:2267] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.842032 1177 consensus_queue.cc:260] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:36.842638 2036 ts_tablet_manager.cc:1905] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.842010 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 8a12b225131b464494a9f02133d37aea (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]) successfully deleted
I20250112 02:52:36.843156 449 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:36.845168 1177 ts_tablet_manager.cc:1428] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250112 02:52:36.850055 2222 raft_consensus.cc:491] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.850165 2222 raft_consensus.cc:513] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.850688 2222 leader_election.cc:290] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:36.851223 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfd1ebe729d458ea31eb7feff7ee072" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:36.851531 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfd1ebe729d458ea31eb7feff7ee072" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:36.851944 854 leader_election.cc:343] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:36.852342 855 leader_election.cc:343] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:36.852444 855 leader_election.cc:304] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:36.853070 2222 raft_consensus.cc:2743] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:36.856703 1173 tablet_bootstrap.cc:492] T b9537fcd96eb41cc8f680075ec71137b P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:36.856812 1173 ts_tablet_manager.cc:1397] T b9537fcd96eb41cc8f680075ec71137b P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.053s user 0.016s sys 0.000s
I20250112 02:52:36.857059 2047 consensus_queue.cc:1035] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:36.857470 1173 raft_consensus.cc:357] T b9537fcd96eb41cc8f680075ec71137b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.857645 1173 raft_consensus.cc:383] T b9537fcd96eb41cc8f680075ec71137b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.857705 1173 raft_consensus.cc:738] T b9537fcd96eb41cc8f680075ec71137b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.857913 1173 consensus_queue.cc:260] T b9537fcd96eb41cc8f680075ec71137b P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.858433 1173 ts_tablet_manager.cc:1428] T b9537fcd96eb41cc8f680075ec71137b P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:36.858692 1173 tablet_bootstrap.cc:492] T c8a7281c0c09442ebd0cc13b8496c86e P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:36.859637 2047 raft_consensus.cc:491] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.859750 2047 raft_consensus.cc:513] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.860267 2047 leader_election.cc:290] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:36.860816 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b184806b4884cd3b0a002905a1e57af" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:36.860962 1051 raft_consensus.cc:2463] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:36.861399 714 leader_election.cc:304] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:36.861982 2047 raft_consensus.cc:2798] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:36.862056 2047 raft_consensus.cc:491] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:36.862115 2047 raft_consensus.cc:3054] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.863131 1173 tablet_bootstrap.cc:654] T c8a7281c0c09442ebd0cc13b8496c86e P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.839269 2003 rpcz_store.cc:276] 0112 02:52:34.243889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.243925 (+ 36us) service_pool.cc:225] Handling call
0112 02:52:36.831881 (+2587956us) ts_tablet_manager.cc:1145] Deleting tablet b7bd96fb381f4e73a0081be3fc799c36
0112 02:52:36.831892 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.835510 (+ 3618us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.838098 (+ 2588us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.839172 (+ 1074us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2587778}
I20250112 02:52:36.868597 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet b7bd96fb381f4e73a0081be3fc799c36 (table default.Table_893bf955f62b495c9dcc59390b472f96 [id=e992deac51c64dd8a022428dd3c53714]) successfully deleted
I20250112 02:52:36.868721 2003 tablet_replica.cc:331] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:36.869434 1173 tablet_bootstrap.cc:492] T c8a7281c0c09442ebd0cc13b8496c86e P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:36.869556 1173 ts_tablet_manager.cc:1397] T c8a7281c0c09442ebd0cc13b8496c86e P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20250112 02:52:36.870272 1173 raft_consensus.cc:357] T c8a7281c0c09442ebd0cc13b8496c86e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.870445 1173 raft_consensus.cc:383] T c8a7281c0c09442ebd0cc13b8496c86e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.870493 1173 raft_consensus.cc:738] T c8a7281c0c09442ebd0cc13b8496c86e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.870719 1173 consensus_queue.cc:260] T c8a7281c0c09442ebd0cc13b8496c86e P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.871292 1173 ts_tablet_manager.cc:1428] T c8a7281c0c09442ebd0cc13b8496c86e P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:36.871628 1173 tablet_bootstrap.cc:492] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:36.871695 2222 raft_consensus.cc:491] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.871946 2222 raft_consensus.cc:513] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.872886 1173 tablet_bootstrap.cc:654] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.873162 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d84027ad6d40a5be5afcdd854431a5" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:36.873574 854 leader_election.cc:343] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:36.873788 2222 leader_election.cc:290] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:36.874154 2003 raft_consensus.cc:2238] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:36.874270 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d84027ad6d40a5be5afcdd854431a5" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:36.874364 2003 pending_rounds.cc:62] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:36.874399 2003 pending_rounds.cc:69] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319860820742144 op_type: NO_OP noop_request { }
I20250112 02:52:36.874465 2003 raft_consensus.cc:2883] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:36.874533 2003 raft_consensus.cc:2267] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:36.874681 855 leader_election.cc:343] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:36.874761 855 leader_election.cc:304] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:36.875061 2222 raft_consensus.cc:2743] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:36.875084 2003 ts_tablet_manager.cc:1905] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.879681 2047 raft_consensus.cc:513] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:36.880921 989 consensus_peers.cc:487] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d5b3df88a0c45dba879b3d506710614. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:36.881887 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b184806b4884cd3b0a002905a1e57af" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:36.882053 1051 raft_consensus.cc:3054] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:36.881033 713 leader_election.cc:343] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 1b184806b4884cd3b0a002905a1e57af
I20250112 02:52:36.886862 1051 raft_consensus.cc:2463] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:36.887362 714 leader_election.cc:304] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:36.888113 2179 raft_consensus.cc:2798] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:36.888221 2179 raft_consensus.cc:695] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:36.888463 2179 consensus_queue.cc:237] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.889294 2003 ts_tablet_manager.cc:1918] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.889370 2003 log.cc:1198] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/ac716c6da01941a989754aac1ef430d5
I20250112 02:52:36.889654 2003 ts_tablet_manager.cc:1939] T ac716c6da01941a989754aac1ef430d5 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:36.890249 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 60) took 2646 ms. Trace:
W20250112 02:52:36.894665 713 leader_election.cc:343] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 1b184806b4884cd3b0a002905a1e57af
I20250112 02:52:36.881626 2047 leader_election.cc:290] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:36.890333 2003 rpcz_store.cc:276] 0112 02:52:34.244263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.244298 (+ 35us) service_pool.cc:225] Handling call
0112 02:52:36.868679 (+2624381us) ts_tablet_manager.cc:1145] Deleting tablet ac716c6da01941a989754aac1ef430d5
0112 02:52:36.868688 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.878152 (+ 9464us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.889213 (+ 11061us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.890213 (+ 1000us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5269,"tablet-delete.queue_time_us":2624248}
I20250112 02:52:36.903734 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet ac716c6da01941a989754aac1ef430d5 (table default.Table_bee5a719ea4b44b78254bae566a0ba92 [id=d3fa1ea0552e4aaeabd6ac3ac84ca389]) successfully deleted
I20250112 02:52:36.903821 2003 tablet_replica.cc:331] T 02d0f67141dc46b6a87ca0ad080d5bfe P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:36.904012 2003 raft_consensus.cc:2238] T 02d0f67141dc46b6a87ca0ad080d5bfe P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.904168 2003 raft_consensus.cc:2267] T 02d0f67141dc46b6a87ca0ad080d5bfe P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.904753 2003 ts_tablet_manager.cc:1905] T 02d0f67141dc46b6a87ca0ad080d5bfe P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.905194 827 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:36.906982 756 tablet_service.cc:1467] Processing CreateTablet for tablet f950c604609042da96e17398f42c1929 (DEFAULT_TABLE table=default.Table_8dde1cf3a26244709cf38e8563b97a2a [id=8b29f4bd13ba4a24bb7f5e98d83354ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.907349 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f950c604609042da96e17398f42c1929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.907373 1017 tablet_service.cc:1467] Processing CreateTablet for tablet f950c604609042da96e17398f42c1929 (DEFAULT_TABLE table=default.Table_8dde1cf3a26244709cf38e8563b97a2a [id=8b29f4bd13ba4a24bb7f5e98d83354ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.907718 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f950c604609042da96e17398f42c1929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.916014 2024 ts_tablet_manager.cc:1918] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.916574 1102 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:36.917982 756 tablet_service.cc:1467] Processing CreateTablet for tablet f29fb4cc96f64cbeabad085f1007716f (DEFAULT_TABLE table=default.Table_2312878f87074a63af9c96478ffb7416 [id=8fb56725c8a341b8a5e76db88275cf17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.918299 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f29fb4cc96f64cbeabad085f1007716f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.918318 1017 tablet_service.cc:1467] Processing CreateTablet for tablet f29fb4cc96f64cbeabad085f1007716f (DEFAULT_TABLE table=default.Table_2312878f87074a63af9c96478ffb7416 [id=8fb56725c8a341b8a5e76db88275cf17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.918628 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f29fb4cc96f64cbeabad085f1007716f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.924321 2024 log.cc:1198] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/3c686fdc8896463c81c9f7c5f53a3da2
I20250112 02:52:36.924273 742 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb24b3d8a9446e8abd9729462225002 (DEFAULT_TABLE table=default.Table_78163902681c4b26885de83c218c408f [id=edab6f55e1ed411e958efc90fc411d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.924618 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb24b3d8a9446e8abd9729462225002. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.924665 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb24b3d8a9446e8abd9729462225002 (DEFAULT_TABLE table=default.Table_78163902681c4b26885de83c218c408f [id=edab6f55e1ed411e958efc90fc411d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.924860 2024 ts_tablet_manager.cc:1939] T 3c686fdc8896463c81c9f7c5f53a3da2 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:36.924968 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb24b3d8a9446e8abd9729462225002. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.929898 2215 raft_consensus.cc:491] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.930022 2215 raft_consensus.cc:513] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.930599 2215 leader_election.cc:290] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:36.931032 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87759e8b99b04e628badc8f60cb38bb3" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:36.931207 778 raft_consensus.cc:2463] T 87759e8b99b04e628badc8f60cb38bb3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:36.931214 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87759e8b99b04e628badc8f60cb38bb3" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:36.931404 910 raft_consensus.cc:2463] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:36.931584 991 leader_election.cc:304] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:36.931856 2050 raft_consensus.cc:2798] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:36.931895 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 51) took 2461 ms. Trace:
I20250112 02:52:36.931926 2050 raft_consensus.cc:491] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:36.931983 2050 raft_consensus.cc:3054] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.931970 2024 rpcz_store.cc:276] 0112 02:52:34.470275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.470314 (+ 39us) service_pool.cc:225] Handling call
0112 02:52:36.823578 (+2353264us) ts_tablet_manager.cc:1145] Deleting tablet 3c686fdc8896463c81c9f7c5f53a3da2
0112 02:52:36.823586 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.913054 (+ 89468us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.915931 (+ 2877us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.931869 (+ 15938us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2352312}
I20250112 02:52:36.932366 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 52) took 2461 ms. Trace:
I20250112 02:52:36.932438 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 3c686fdc8896463c81c9f7c5f53a3da2 (table default.Table_4934f6022f2441eda88751236a5798be [id=5c4769bfcefd4fe0a0f3c2e315d50c32]) successfully deleted
I20250112 02:52:36.932452 2024 rpcz_store.cc:276] 0112 02:52:34.470562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.470600 (+ 38us) service_pool.cc:225] Handling call
0112 02:52:36.932251 (+2461651us) ts_tablet_manager.cc:1145] Deleting tablet d63d411f906b4d4eb68343bdcc493e05
0112 02:52:36.932262 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.932351 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2460902}
I20250112 02:52:36.932715 2024 tablet_replica.cc:331] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:36.932902 2024 raft_consensus.cc:2238] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
W20250112 02:52:36.933072 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet d63d411f906b4d4eb68343bdcc493e05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d63d411f906b4d4eb68343bdcc493e05 already in progress: creating tablet
I20250112 02:52:36.933220 2024 pending_rounds.cc:62] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6: Trying to abort 1 pending ops.
I20250112 02:52:36.933264 2024 pending_rounds.cc:69] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319831906189312 op_type: NO_OP noop_request { }
I20250112 02:52:36.933363 2024 raft_consensus.cc:2883] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:36.933449 2024 raft_consensus.cc:2267] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.934089 2024 ts_tablet_manager.cc:1905] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.934355 2050 raft_consensus.cc:513] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.935071 2050 leader_election.cc:290] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:36.935688 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87759e8b99b04e628badc8f60cb38bb3" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:36.935843 778 raft_consensus.cc:3054] T 87759e8b99b04e628badc8f60cb38bb3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.935878 2003 ts_tablet_manager.cc:1918] T 02d0f67141dc46b6a87ca0ad080d5bfe P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.935952 2003 log.cc:1198] T 02d0f67141dc46b6a87ca0ad080d5bfe P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/02d0f67141dc46b6a87ca0ad080d5bfe
I20250112 02:52:36.936265 2003 ts_tablet_manager.cc:1939] T 02d0f67141dc46b6a87ca0ad080d5bfe P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:36.936547 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 62) took 2691 ms. Trace:
I20250112 02:52:36.936599 742 tablet_service.cc:1467] Processing CreateTablet for tablet a243d008c17a46668a3d40e015e84f11 (DEFAULT_TABLE table=default.Table_c586400629b14725a93d61c2c6e4ad82 [id=c5fa895ef53d4153a152b25898dded3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.936903 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a243d008c17a46668a3d40e015e84f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.938405 778 raft_consensus.cc:2463] T 87759e8b99b04e628badc8f60cb38bb3 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:36.938805 991 leader_election.cc:304] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:36.939093 2050 raft_consensus.cc:2798] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:36.939188 2050 raft_consensus.cc:695] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:36.939374 2050 consensus_queue.cc:237] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.939600 1034 tablet_service.cc:1467] Processing CreateTablet for tablet a243d008c17a46668a3d40e015e84f11 (DEFAULT_TABLE table=default.Table_c586400629b14725a93d61c2c6e4ad82 [id=c5fa895ef53d4153a152b25898dded3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.940755 756 tablet_service.cc:1467] Processing CreateTablet for tablet bcfa2b316ddd48faa0dc2dd8db8690f9 (DEFAULT_TABLE table=default.Table_635bfb7ffd504fb89c9340cdf227944d [id=e125388e82274e918ba88fcaf3881d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.941046 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcfa2b316ddd48faa0dc2dd8db8690f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.941100 1032 tablet_service.cc:1467] Processing CreateTablet for tablet bcfa2b316ddd48faa0dc2dd8db8690f9 (DEFAULT_TABLE table=default.Table_635bfb7ffd504fb89c9340cdf227944d [id=e125388e82274e918ba88fcaf3881d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.945101 755 tablet_service.cc:1467] Processing CreateTablet for tablet d0cb54003a83472cb9c57ace3e65b0be (DEFAULT_TABLE table=default.Table_cb4ed576320c497588e65dcbac70afe9 [id=eee3e8b1198e401c93b8016a7c69b525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.945384 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0cb54003a83472cb9c57ace3e65b0be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.945981 2215 raft_consensus.cc:491] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.946087 2215 raft_consensus.cc:513] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.946580 2215 leader_election.cc:290] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:36.947062 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1811ce8982fd4003a35e89ea34e32746" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:36.947279 778 raft_consensus.cc:2463] T 1811ce8982fd4003a35e89ea34e32746 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:36.947718 1031 tablet_service.cc:1467] Processing CreateTablet for tablet d0cb54003a83472cb9c57ace3e65b0be (DEFAULT_TABLE table=default.Table_cb4ed576320c497588e65dcbac70afe9 [id=eee3e8b1198e401c93b8016a7c69b525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.935436 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87759e8b99b04e628badc8f60cb38bb3" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:36.946977 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1811ce8982fd4003a35e89ea34e32746" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:36.948266 910 raft_consensus.cc:3054] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.948349 912 raft_consensus.cc:2463] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:36.947726 991 leader_election.cc:304] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:36.948483 898 tablet_service.cc:1467] Processing CreateTablet for tablet f950c604609042da96e17398f42c1929 (DEFAULT_TABLE table=default.Table_8dde1cf3a26244709cf38e8563b97a2a [id=8b29f4bd13ba4a24bb7f5e98d83354ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.948714 2215 raft_consensus.cc:2798] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:36.948772 2215 raft_consensus.cc:491] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:36.948807 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f950c604609042da96e17398f42c1929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.948824 2215 raft_consensus.cc:3054] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.950915 2215 raft_consensus.cc:513] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.951836 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1811ce8982fd4003a35e89ea34e32746" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:36.951987 778 raft_consensus.cc:3054] T 1811ce8982fd4003a35e89ea34e32746 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.952538 899 tablet_service.cc:1467] Processing CreateTablet for tablet f29fb4cc96f64cbeabad085f1007716f (DEFAULT_TABLE table=default.Table_2312878f87074a63af9c96478ffb7416 [id=8fb56725c8a341b8a5e76db88275cf17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.952816 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f29fb4cc96f64cbeabad085f1007716f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.953166 889 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb24b3d8a9446e8abd9729462225002 (DEFAULT_TABLE table=default.Table_78163902681c4b26885de83c218c408f [id=edab6f55e1ed411e958efc90fc411d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.953442 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb24b3d8a9446e8abd9729462225002. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.954778 2381 sys_catalog.cc:263] Verifying existing consensus state
I20250112 02:52:36.955320 1177 tablet_bootstrap.cc:492] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:36.956918 2215 leader_election.cc:290] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:36.958444 778 raft_consensus.cc:2463] T 1811ce8982fd4003a35e89ea34e32746 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:36.959198 910 raft_consensus.cc:2463] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:36.959970 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1811ce8982fd4003a35e89ea34e32746" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:36.960107 910 raft_consensus.cc:3054] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:36.961301 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcfa2b316ddd48faa0dc2dd8db8690f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.963692 991 leader_election.cc:304] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:36.963917 2215 raft_consensus.cc:2798] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:36.963865 890 tablet_service.cc:1467] Processing CreateTablet for tablet a243d008c17a46668a3d40e015e84f11 (DEFAULT_TABLE table=default.Table_c586400629b14725a93d61c2c6e4ad82 [id=c5fa895ef53d4153a152b25898dded3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.964077 2215 raft_consensus.cc:695] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:36.964530 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a243d008c17a46668a3d40e015e84f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.964776 2215 consensus_queue.cc:237] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.936606 2003 rpcz_store.cc:276] 0112 02:52:34.244921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.244970 (+ 49us) service_pool.cc:225] Handling call
0112 02:52:36.903775 (+2658805us) ts_tablet_manager.cc:1145] Deleting tablet 02d0f67141dc46b6a87ca0ad080d5bfe
0112 02:52:36.903785 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.925619 (+ 21834us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.935808 (+ 10189us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.936535 (+ 727us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2658604}
I20250112 02:52:36.966972 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 02d0f67141dc46b6a87ca0ad080d5bfe (table default.Table_3a84890c771c4574a25d33543fe7217d [id=49a61d0f688b488da9a139f0a64d2834]) successfully deleted
I20250112 02:52:36.964334 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0cb54003a83472cb9c57ace3e65b0be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.970104 2003 tablet_replica.cc:331] T ea33e5c1a1fb46599c5836a36b130051 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:36.970259 2003 raft_consensus.cc:2238] T ea33e5c1a1fb46599c5836a36b130051 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.970371 2003 raft_consensus.cc:2267] T ea33e5c1a1fb46599c5836a36b130051 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.970794 2003 ts_tablet_manager.cc:1905] T ea33e5c1a1fb46599c5836a36b130051 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.972971 2036 ts_tablet_manager.cc:1918] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:36.973058 2036 log.cc:1198] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/d4abf59d96414e5faa7cf75ba133832c
I20250112 02:52:36.973302 2036 ts_tablet_manager.cc:1939] T d4abf59d96414e5faa7cf75ba133832c P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:36.973865 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 61) took 2726 ms. Trace:
I20250112 02:52:36.973927 2036 rpcz_store.cc:276] 0112 02:52:34.247785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.277869 (+ 30084us) service_pool.cc:225] Handling call
0112 02:52:36.841741 (+2563872us) ts_tablet_manager.cc:1145] Deleting tablet d4abf59d96414e5faa7cf75ba133832c
0112 02:52:36.841750 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.953265 (+111515us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.972894 (+ 19629us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.973849 (+ 955us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2563743}
I20250112 02:52:36.974215 2036 tablet_replica.cc:331] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:36.974435 2036 raft_consensus.cc:2238] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.974572 2036 raft_consensus.cc:2267] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.975441 1177 tablet_bootstrap.cc:654] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.975775 2024 ts_tablet_manager.cc:1918] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.975859 2024 log.cc:1198] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/6b521522c4784d2ab4056268823f1099
I20250112 02:52:36.976115 2024 ts_tablet_manager.cc:1939] T 6b521522c4784d2ab4056268823f1099 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:36.978399 2003 ts_tablet_manager.cc:1918] T ea33e5c1a1fb46599c5836a36b130051 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:36.978724 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 53) took 2507 ms. Trace:
I20250112 02:52:36.978829 2024 rpcz_store.cc:276] 0112 02:52:34.470835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.470874 (+ 39us) service_pool.cc:225] Handling call
0112 02:52:36.932674 (+2461800us) ts_tablet_manager.cc:1145] Deleting tablet 6b521522c4784d2ab4056268823f1099
0112 02:52:36.932684 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.938381 (+ 5697us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.975713 (+ 37332us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.978712 (+ 2999us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2461292}
I20250112 02:52:36.979269 2024 tablet_replica.cc:331] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:36.979218 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 6b521522c4784d2ab4056268823f1099 (table default.Table_00028054ce7c431b96d17d8f6f8856a7 [id=f8cd4289f89a4a748c26a7a01236406e]) successfully deleted
I20250112 02:52:36.979676 1181 tablet_bootstrap.cc:654] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:36.979688 2024 raft_consensus.cc:2238] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:36.980013 2024 raft_consensus.cc:2267] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.980141 2381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Bootstrap starting.
I20250112 02:52:36.980523 2024 ts_tablet_manager.cc:1905] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.980610 1173 tablet_bootstrap.cc:492] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:36.980728 1173 ts_tablet_manager.cc:1397] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.109s user 0.008s sys 0.000s
I20250112 02:52:36.981385 1173 raft_consensus.cc:357] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:36.981556 1173 raft_consensus.cc:383] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.981618 1173 raft_consensus.cc:738] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.981776 2003 log.cc:1198] T ea33e5c1a1fb46599c5836a36b130051 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/ea33e5c1a1fb46599c5836a36b130051
I20250112 02:52:36.981812 1173 consensus_queue.cc:260] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:36.982425 1173 ts_tablet_manager.cc:1428] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:36.982702 1173 tablet_bootstrap.cc:492] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:36.983232 2036 ts_tablet_manager.cc:1905] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.982453 2003 ts_tablet_manager.cc:1939] T ea33e5c1a1fb46599c5836a36b130051 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:36.983651 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 64) took 2738 ms. Trace:
I20250112 02:52:36.984426 892 tablet_service.cc:1467] Processing CreateTablet for tablet bcfa2b316ddd48faa0dc2dd8db8690f9 (DEFAULT_TABLE table=default.Table_635bfb7ffd504fb89c9340cdf227944d [id=e125388e82274e918ba88fcaf3881d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.984737 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcfa2b316ddd48faa0dc2dd8db8690f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.983765 2003 rpcz_store.cc:276] 0112 02:52:34.245643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.245688 (+ 45us) service_pool.cc:225] Handling call
0112 02:52:36.966696 (+2721008us) ts_tablet_manager.cc:1145] Deleting tablet ea33e5c1a1fb46599c5836a36b130051
0112 02:52:36.970072 (+ 3376us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.973960 (+ 3888us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.978337 (+ 4377us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:36.983638 (+ 5301us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2720847}
I20250112 02:52:36.985134 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet ea33e5c1a1fb46599c5836a36b130051 (table default.Table_7abcf14838c744109a08985ba6ce3e0d [id=5f25c42be8924b41be7f2f268d068e04]) successfully deleted
I20250112 02:52:36.985203 2003 tablet_replica.cc:331] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:36.985380 2003 raft_consensus.cc:2238] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:36.985569 2003 raft_consensus.cc:2267] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:36.986110 2003 ts_tablet_manager.cc:1905] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:36.988080 1177 tablet_bootstrap.cc:492] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:36.988193 1177 ts_tablet_manager.cc:1397] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.143s user 0.002s sys 0.000s
I20250112 02:52:36.989071 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a243d008c17a46668a3d40e015e84f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.988926 1177 raft_consensus.cc:357] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:36.989323 1177 raft_consensus.cc:383] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:36.989727 891 tablet_service.cc:1467] Processing CreateTablet for tablet d0cb54003a83472cb9c57ace3e65b0be (DEFAULT_TABLE table=default.Table_cb4ed576320c497588e65dcbac70afe9 [id=eee3e8b1198e401c93b8016a7c69b525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:36.989820 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet d4abf59d96414e5faa7cf75ba133832c (table default.Table_525750dad2b7461586d070dbaa74d0b1 [id=7d048aa907a74605964b68726ac5d759]) successfully deleted
I20250112 02:52:36.990043 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0cb54003a83472cb9c57ace3e65b0be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:36.990015 2222 raft_consensus.cc:491] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:36.990115 2222 raft_consensus.cc:513] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:36.990703 2222 leader_election.cc:290] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:36.995047 1177 raft_consensus.cc:738] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:36.995265 1177 consensus_queue.cc:260] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:36.996050 1177 ts_tablet_manager.cc:1428] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20250112 02:52:36.996156 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d659c5a4f7a446c6aaf2283f65b45106" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:36.996320 775 raft_consensus.cc:2463] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
W20250112 02:52:36.995364 854 leader_election.cc:343] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: d659c5a4f7a446c6aaf2283f65b45106
I20250112 02:52:37.000762 1177 tablet_bootstrap.cc:492] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:36.993149 2228 raft_consensus.cc:491] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.001168 2228 raft_consensus.cc:513] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.001725 2228 leader_election.cc:290] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:37.003943 991 leader_election.cc:343] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: f7cd0927c6a4401cbe69f55d4db9963a
I20250112 02:52:37.004171 1173 tablet_bootstrap.cc:654] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.008051 2222 raft_consensus.cc:491] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.008158 2222 raft_consensus.cc:513] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.009536 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7cd0927c6a4401cbe69f55d4db9963a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:37.009698 919 raft_consensus.cc:2463] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:37.010021 989 leader_election.cc:304] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:37.010421 2228 raft_consensus.cc:2798] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:37.010485 2228 raft_consensus.cc:491] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:37.010532 2228 raft_consensus.cc:3054] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.013221 2228 raft_consensus.cc:513] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:37.014683 991 leader_election.cc:343] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: f7cd0927c6a4401cbe69f55d4db9963a
I20250112 02:52:37.015457 2228 leader_election.cc:290] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:37.015938 1177 tablet_bootstrap.cc:654] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.019202 855 leader_election.cc:304] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:37.019407 2368 raft_consensus.cc:2798] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 02:52:37.019488 2368 raft_consensus.cc:491] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:37.019554 2368 raft_consensus.cc:3054] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Advancing to term 2
I20250112 02:52:37.020669 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e67f0a0c065498f9a644220414415b0" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:37.021165 854 leader_election.cc:343] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.022095 2036 ts_tablet_manager.cc:1918] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.022209 2036 log.cc:1198] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/c4a3486ef011492c8a1df8d6278adedf
I20250112 02:52:37.025764 910 raft_consensus.cc:2463] T 1811ce8982fd4003a35e89ea34e32746 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:37.026244 2024 ts_tablet_manager.cc:1918] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.026386 2024 log.cc:1198] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/49bb6d20cfa0453e90e048825114525e
I20250112 02:52:37.026783 2024 ts_tablet_manager.cc:1939] T 49bb6d20cfa0453e90e048825114525e P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:37.027072 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 54) took 2556 ms. Trace:
I20250112 02:52:37.027181 2024 rpcz_store.cc:276] 0112 02:52:34.471061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.476762 (+ 5701us) service_pool.cc:225] Handling call
0112 02:52:36.979229 (+2502467us) ts_tablet_manager.cc:1145] Deleting tablet 49bb6d20cfa0453e90e048825114525e
0112 02:52:36.979238 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.994264 (+ 15026us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.026177 (+ 31913us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.027059 (+ 882us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2502274}
I20250112 02:52:37.027700 2024 tablet_replica.cc:331] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:37.027921 2024 raft_consensus.cc:2238] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.028087 2024 raft_consensus.cc:2267] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.028589 2024 ts_tablet_manager.cc:1905] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.028602 1173 tablet_bootstrap.cc:492] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:37.029314 1173 ts_tablet_manager.cc:1397] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.000s
I20250112 02:52:37.029906 1173 raft_consensus.cc:357] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.030076 1173 raft_consensus.cc:383] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.030125 1173 raft_consensus.cc:738] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.030280 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 49bb6d20cfa0453e90e048825114525e (table default.Table_f359341b73c14c60b859f06dc5f428a7 [id=bedc9786ee5b40a9bdcbd3cd0be38361]) successfully deleted
I20250112 02:52:37.030813 1177 tablet_bootstrap.cc:492] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:37.020226 2222 leader_election.cc:290] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:37.032132 2003 ts_tablet_manager.cc:1918] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.032245 2003 log.cc:1198] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/fb0aba0139cf479b80b32f3dfc936535
I20250112 02:52:37.032629 2003 ts_tablet_manager.cc:1939] T fb0aba0139cf479b80b32f3dfc936535 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:37.015530 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7cd0927c6a4401cbe69f55d4db9963a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:37.033748 919 raft_consensus.cc:3054] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.034029 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e67f0a0c065498f9a644220414415b0" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:37.034554 855 leader_election.cc:343] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.034660 855 leader_election.cc:304] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:37.034853 2222 raft_consensus.cc:2743] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:37.036175 1177 ts_tablet_manager.cc:1397] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.003s
I20250112 02:52:37.036854 1177 raft_consensus.cc:357] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.037011 1177 raft_consensus.cc:383] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.037055 1177 raft_consensus.cc:738] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.037217 1177 consensus_queue.cc:260] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.037698 1177 ts_tablet_manager.cc:1428] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:37.037950 1177 tablet_bootstrap.cc:492] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:37.039436 1177 tablet_bootstrap.cc:654] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.043424 2036 ts_tablet_manager.cc:1939] T c4a3486ef011492c8a1df8d6278adedf P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.044822 2050 consensus_queue.cc:1035] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:37.049171 1173 consensus_queue.cc:260] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.049645 1173 ts_tablet_manager.cc:1428] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.020s user 0.001s sys 0.000s
I20250112 02:52:37.049850 1173 tablet_bootstrap.cc:492] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:37.051157 1173 tablet_bootstrap.cc:654] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.053071 2024 ts_tablet_manager.cc:1918] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.058488 2179 raft_consensus.cc:491] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.058468 2368 raft_consensus.cc:513] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.058595 2179 raft_consensus.cc:513] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.058988 2368 leader_election.cc:290] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 2 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:37.059113 2179 leader_election.cc:290] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:37.059412 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d659c5a4f7a446c6aaf2283f65b45106" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:37.059576 774 raft_consensus.cc:3054] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Advancing to term 2
W20250112 02:52:37.060693 854 leader_election.cc:343] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: d659c5a4f7a446c6aaf2283f65b45106
I20250112 02:52:37.061987 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:37.062114 774 raft_consensus.cc:2463] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 2.
I20250112 02:52:37.064380 1177 tablet_bootstrap.cc:492] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:37.064502 1177 ts_tablet_manager.cc:1397] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.003s
I20250112 02:52:37.064517 919 raft_consensus.cc:2463] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:37.066028 1177 raft_consensus.cc:357] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.066234 1177 raft_consensus.cc:383] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.066300 1177 raft_consensus.cc:738] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.066658 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:37.067544 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b64e4e64cc462891f1ee7a594d5026" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:37.067627 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 62) took 2813 ms. Trace:
I20250112 02:52:37.067894 2036 rpcz_store.cc:276] 0112 02:52:34.247880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.277998 (+ 30118us) service_pool.cc:225] Handling call
0112 02:52:36.974167 (+2696169us) ts_tablet_manager.cc:1145] Deleting tablet c4a3486ef011492c8a1df8d6278adedf
0112 02:52:36.974177 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.007677 (+ 33500us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.022025 (+ 14348us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.061591 (+ 39566us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2696040}
I20250112 02:52:37.068702 2036 tablet_replica.cc:331] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.068892 2036 raft_consensus.cc:2238] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.069046 2036 raft_consensus.cc:2267] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.069553 2036 ts_tablet_manager.cc:1905] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.069938 910 raft_consensus.cc:2463] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:37.070276 713 leader_election.cc:304] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:37.070554 2179 raft_consensus.cc:2798] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:37.070618 2179 raft_consensus.cc:491] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:37.070664 2179 raft_consensus.cc:3054] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.067382 1177 consensus_queue.cc:260] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.072182 1177 ts_tablet_manager.cc:1428] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250112 02:52:37.072449 1177 tablet_bootstrap.cc:492] T b3c5a861a90840d0a33b780ec75bffd2 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:37.072901 2179 raft_consensus.cc:513] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.073932 1177 tablet_bootstrap.cc:654] T b3c5a861a90840d0a33b780ec75bffd2 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.074098 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 65) took 2828 ms. Trace:
I20250112 02:52:37.074190 2003 rpcz_store.cc:276] 0112 02:52:34.246010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.246051 (+ 41us) service_pool.cc:225] Handling call
0112 02:52:36.985168 (+2739117us) ts_tablet_manager.cc:1145] Deleting tablet fb0aba0139cf479b80b32f3dfc936535
0112 02:52:36.985176 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:36.989413 (+ 4237us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.022596 (+ 33183us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.074082 (+ 51486us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2738917}
I20250112 02:52:37.074586 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 67) took 2828 ms. Trace:
I20250112 02:52:37.074658 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet fb0aba0139cf479b80b32f3dfc936535 (table default.Table_672d7452ad6d4c99ae9f2ae77b592f78 [id=33dad7ab4106404aaf33266ed1ea5635]) successfully deleted
I20250112 02:52:37.074661 2003 rpcz_store.cc:276] 0112 02:52:34.246571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.246623 (+ 52us) service_pool.cc:225] Handling call
0112 02:52:37.074465 (+2827842us) ts_tablet_manager.cc:1145] Deleting tablet f05c8426ea3d4fe89fbe6c88c688019f
0112 02:52:37.074475 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.074572 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2827656}
I20250112 02:52:37.075080 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 68) took 2828 ms. Trace:
I20250112 02:52:37.075158 2003 rpcz_store.cc:276] 0112 02:52:34.246699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.246744 (+ 45us) service_pool.cc:225] Handling call
0112 02:52:37.074990 (+2828246us) ts_tablet_manager.cc:1145] Deleting tablet daef3a58605e40978f7389698b22265d
0112 02:52:37.075001 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.075064 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2828053}
I20250112 02:52:37.075414 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 69) took 2828 ms. Trace:
W20250112 02:52:37.075392 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f05c8426ea3d4fe89fbe6c88c688019f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f05c8426ea3d4fe89fbe6c88c688019f already in progress: creating tablet
I20250112 02:52:37.075783 2179 leader_election.cc:290] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:37.076156 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet daef3a58605e40978f7389698b22265d because the tablet was not found. No further retry: Not found: Tablet not found: daef3a58605e40978f7389698b22265d
I20250112 02:52:37.076810 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:37.078991 965 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:37.079437 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:37.080592 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
I20250112 02:52:37.081324 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet f3771166b77e4f90aaa116ab1a9dc299 (table default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6]). Expected version 1 got 0
I20250112 02:52:37.082005 2003 rpcz_store.cc:276] 0112 02:52:34.246812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.246880 (+ 68us) service_pool.cc:225] Handling call
0112 02:52:37.075367 (+2828487us) ts_tablet_manager.cc:1145] Deleting tablet b2ccb88108f24e36ba4b7ebf36398950
0112 02:52:37.075372 (+ 5us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.075406 (+ 34us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2828315}
I20250112 02:52:37.083112 1181 tablet_bootstrap.cc:492] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:37.083226 1181 ts_tablet_manager.cc:1397] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.304s user 0.014s sys 0.006s
I20250112 02:52:37.083942 1181 raft_consensus.cc:357] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.084084 1181 raft_consensus.cc:383] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.084120 1181 raft_consensus.cc:738] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.084254 1181 consensus_queue.cc:260] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.084667 1181 ts_tablet_manager.cc:1428] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250112 02:52:37.084715 2036 ts_tablet_manager.cc:1918] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.084798 2036 log.cc:1198] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/cc62dcbf67e84e36ae1fbe127ccd4879
I20250112 02:52:37.084875 1181 tablet_bootstrap.cc:492] T a437952889494b0e820e41fdfffe708d P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:37.085096 2036 ts_tablet_manager.cc:1939] T cc62dcbf67e84e36ae1fbe127ccd4879 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.085386 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 63) took 2837 ms. Trace:
I20250112 02:52:37.085454 2036 rpcz_store.cc:276] 0112 02:52:34.247973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.278137 (+ 30164us) service_pool.cc:225] Handling call
0112 02:52:37.068659 (+2790522us) ts_tablet_manager.cc:1145] Deleting tablet cc62dcbf67e84e36ae1fbe127ccd4879
0112 02:52:37.068668 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.081012 (+ 12344us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.084647 (+ 3635us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.085372 (+ 725us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2790423}
I20250112 02:52:37.085729 2036 tablet_replica.cc:331] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.085903 2036 raft_consensus.cc:2238] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.086024 2036 raft_consensus.cc:2267] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.086277 1181 tablet_bootstrap.cc:654] T a437952889494b0e820e41fdfffe708d P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.086406 2036 ts_tablet_manager.cc:1905] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.087072 966 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:37.092363 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 70) took 2845 ms. Trace:
I20250112 02:52:37.092470 2003 rpcz_store.cc:276] 0112 02:52:34.246944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.246983 (+ 39us) service_pool.cc:225] Handling call
0112 02:52:37.085204 (+2838221us) ts_tablet_manager.cc:1145] Deleting tablet b6d84027ad6d40a5be5afcdd854431a5
0112 02:52:37.092237 (+ 7033us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.092344 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2838013}
I20250112 02:52:37.091459 2036 ts_tablet_manager.cc:1918] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.092805 2036 log.cc:1198] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/d659c5a4f7a446c6aaf2283f65b45106
I20250112 02:52:37.092876 2003 tablet_replica.cc:331] T 4b599e983ec04f53a440d1ab828d12ab P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:37.093034 2036 ts_tablet_manager.cc:1939] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.093075 2003 raft_consensus.cc:2238] T 4b599e983ec04f53a440d1ab828d12ab P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.060513 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b64e4e64cc462891f1ee7a594d5026" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:37.063509 1173 tablet_bootstrap.cc:492] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:37.093217 2003 raft_consensus.cc:2267] T 4b599e983ec04f53a440d1ab828d12ab P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.093302 1049 raft_consensus.cc:2463] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:37.093371 1173 ts_tablet_manager.cc:1397] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.003s
I20250112 02:52:37.093902 1173 raft_consensus.cc:357] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.094040 1173 raft_consensus.cc:383] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.094074 1173 raft_consensus.cc:738] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.094249 1173 consensus_queue.cc:260] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.094673 1173 ts_tablet_manager.cc:1428] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 02:52:37.094712 2003 ts_tablet_manager.cc:1905] T 4b599e983ec04f53a440d1ab828d12ab P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.094887 1173 tablet_bootstrap.cc:492] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:37.095155 2024 log.cc:1198] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/f6803d0f553b4204b415b2734286bf4a
I20250112 02:52:37.095458 2024 ts_tablet_manager.cc:1939] T f6803d0f553b4204b415b2734286bf4a P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:37.096156 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 55) took 2618 ms. Trace:
I20250112 02:52:37.073859 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b64e4e64cc462891f1ee7a594d5026" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:37.096601 1051 raft_consensus.cc:3054] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.096567 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b64e4e64cc462891f1ee7a594d5026" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:37.096704 910 raft_consensus.cc:3054] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.098821 1051 raft_consensus.cc:2463] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:37.099246 2024 rpcz_store.cc:276] 0112 02:52:34.477577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.477770 (+ 193us) service_pool.cc:225] Handling call
0112 02:52:37.027659 (+2549889us) ts_tablet_manager.cc:1145] Deleting tablet f6803d0f553b4204b415b2734286bf4a
0112 02:52:37.027667 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.050074 (+ 22407us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.052989 (+ 2915us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.096142 (+ 43153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2549688}
I20250112 02:52:37.099779 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 56) took 2621 ms. Trace:
I20250112 02:52:37.100111 2024 rpcz_store.cc:276] 0112 02:52:34.478119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.478533 (+ 414us) service_pool.cc:225] Handling call
0112 02:52:37.099689 (+2621156us) ts_tablet_manager.cc:1145] Deleting tablet c4a3486ef011492c8a1df8d6278adedf
0112 02:52:37.099696 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.099766 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2621021}
W20250112 02:52:37.085402 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b2ccb88108f24e36ba4b7ebf36398950 because the tablet was not found. No further retry: Not found: Tablet not found: b2ccb88108f24e36ba4b7ebf36398950
I20250112 02:52:37.100543 2024 tablet_replica.cc:331] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
W20250112 02:52:37.100615 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b6d84027ad6d40a5be5afcdd854431a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6d84027ad6d40a5be5afcdd854431a5 already in progress: creating tablet
I20250112 02:52:37.101850 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 64) took 2853 ms. Trace:
I20250112 02:52:37.101935 2036 rpcz_store.cc:276] 0112 02:52:34.248053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.278244 (+ 30191us) service_pool.cc:225] Handling call
0112 02:52:37.085693 (+2807449us) ts_tablet_manager.cc:1145] Deleting tablet d659c5a4f7a446c6aaf2283f65b45106
0112 02:52:37.085700 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.089109 (+ 3409us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.091397 (+ 2288us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.101837 (+ 10440us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2807347}
I20250112 02:52:37.096335 1173 tablet_bootstrap.cc:654] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.064975 989 leader_election.cc:304] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:37.102169 714 leader_election.cc:304] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:37.102325 2036 tablet_replica.cc:331] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.102521 2036 raft_consensus.cc:2238] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.102695 2036 raft_consensus.cc:2267] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.102815 2215 raft_consensus.cc:2798] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:37.102826 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet f6803d0f553b4204b415b2734286bf4a (table default.Table_c9cd1883fedc45e89202fd92fbbd0e50 [id=944bc8032e5b4d32a6d568676f53a401]) successfully deleted
I20250112 02:52:37.102916 2215 raft_consensus.cc:695] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
W20250112 02:52:37.103044 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet c4a3486ef011492c8a1df8d6278adedf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4a3486ef011492c8a1df8d6278adedf already in progress: creating tablet
I20250112 02:52:37.103109 2215 consensus_queue.cc:237] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.103111 2036 ts_tablet_manager.cc:1905] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.103672 2024 raft_consensus.cc:2238] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.103780 2024 raft_consensus.cc:2267] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.103914 2047 raft_consensus.cc:2798] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:37.104012 2047 raft_consensus.cc:695] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:37.104154 2024 ts_tablet_manager.cc:1905] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.104195 2047 consensus_queue.cc:237] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.104698 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:37.105046 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet fccff64af8ca4400afc116db98918119 (table default.Table_15bcb2c0f2a74458a1bed231fd3437fc [id=ca9c8281bf254f248076cb600e7c4f87]). Expected version 2 got 0
I20250112 02:52:37.105201 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet f924459d1150472aafb325a11825db23 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]). Expected version 3 got 0
I20250112 02:52:37.105414 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_5e7392cb5a68412f97bddb2677986918 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 1 got 0
I20250112 02:52:37.105718 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:37.106549 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 6117dc8b32a64535a2f4167a03bed0d2 (table default.Table_365739770a11434dbf260ea912e7ea81 [id=868adb627654450c936ef198e49d853b]). Expected version 1 got 0
I20250112 02:52:37.106978 2326 raft_consensus.cc:491] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.107107 2326 raft_consensus.cc:513] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.107435 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:37.107736 2326 leader_election.cc:290] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:37.108758 989 leader_election.cc:343] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: d4abf59d96414e5faa7cf75ba133832c
I20250112 02:52:37.108508 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4abf59d96414e5faa7cf75ba133832c" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:37.108904 775 raft_consensus.cc:2463] T d4abf59d96414e5faa7cf75ba133832c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:37.109622 2056 raft_consensus.cc:491] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.109750 991 leader_election.cc:304] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:37.109735 2056 raft_consensus.cc:513] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.110101 2208 raft_consensus.cc:2798] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:37.110163 2208 raft_consensus.cc:491] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:37.110214 2056 leader_election.cc:290] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:37.110206 2208 raft_consensus.cc:3054] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.110414 1102 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:37.110699 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f234256b343f447dadfc3d1936a824ee" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:37.111035 991 leader_election.cc:343] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:37.111963 989 leader_election.cc:343] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: f234256b343f447dadfc3d1936a824ee
I20250112 02:52:37.112061 989 leader_election.cc:304] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:37.112382 2326 raft_consensus.cc:2743] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:37.112957 2208 raft_consensus.cc:513] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.113389 2208 leader_election.cc:290] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:37.113821 827 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:37.113857 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4abf59d96414e5faa7cf75ba133832c" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
W20250112 02:52:37.113960 989 leader_election.cc:343] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: d4abf59d96414e5faa7cf75ba133832c
I20250112 02:52:37.114100 775 raft_consensus.cc:3054] T d4abf59d96414e5faa7cf75ba133832c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.116825 775 raft_consensus.cc:2463] T d4abf59d96414e5faa7cf75ba133832c P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:37.117178 991 leader_election.cc:304] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:37.117650 2228 raft_consensus.cc:2798] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:37.117769 2228 raft_consensus.cc:695] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:37.117969 2228 consensus_queue.cc:237] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.120194 2024 ts_tablet_manager.cc:1918] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.120414 2024 log.cc:1198] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/97b31ebe035c4552baaffc3133ec73c1
I20250112 02:52:37.120792 2024 ts_tablet_manager.cc:1939] T 97b31ebe035c4552baaffc3133ec73c1 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:37.121101 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 57) took 2642 ms. Trace:
I20250112 02:52:37.121189 2024 rpcz_store.cc:276] 0112 02:52:34.478795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.479043 (+ 248us) service_pool.cc:225] Handling call
0112 02:52:37.100505 (+2621462us) ts_tablet_manager.cc:1145] Deleting tablet 97b31ebe035c4552baaffc3133ec73c1
0112 02:52:37.100514 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.111543 (+ 11029us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.120103 (+ 8560us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.121084 (+ 981us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2621275}
I20250112 02:52:37.121635 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 97b31ebe035c4552baaffc3133ec73c1 (table default.Table_4e7e1a3381d2412987a7061d9030f9f9 [id=2cee6e60fe7e454ab6a9f24e2f5c47cc]) successfully deleted
I20250112 02:52:37.123276 2024 tablet_replica.cc:331] T 64707ee432fb4e96a3bbf7252dfa1813 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:37.123667 2024 raft_consensus.cc:2238] T 64707ee432fb4e96a3bbf7252dfa1813 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.124131 2024 raft_consensus.cc:2267] T 64707ee432fb4e96a3bbf7252dfa1813 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.124780 2024 ts_tablet_manager.cc:1905] T 64707ee432fb4e96a3bbf7252dfa1813 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.125296 2003 ts_tablet_manager.cc:1918] T 4b599e983ec04f53a440d1ab828d12ab P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.127655 2003 log.cc:1198] T 4b599e983ec04f53a440d1ab828d12ab P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/4b599e983ec04f53a440d1ab828d12ab
I20250112 02:52:37.128023 2003 ts_tablet_manager.cc:1939] T 4b599e983ec04f53a440d1ab828d12ab P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:37.128315 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 71) took 2881 ms. Trace:
I20250112 02:52:37.128400 2003 rpcz_store.cc:276] 0112 02:52:34.247050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.247248 (+ 198us) service_pool.cc:225] Handling call
0112 02:52:37.092834 (+2845586us) ts_tablet_manager.cc:1145] Deleting tablet 4b599e983ec04f53a440d1ab828d12ab
0112 02:52:37.092844 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.098705 (+ 5861us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.125218 (+ 26513us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.128299 (+ 3081us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2845407}
I20250112 02:52:37.128820 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 4b599e983ec04f53a440d1ab828d12ab (table default.Table_feacb3254bbe4765a4e0a4908fbc694c [id=f396d2dc8dbd47729f49f7511467d481]) successfully deleted
I20250112 02:52:37.132817 2024 ts_tablet_manager.cc:1918] T 64707ee432fb4e96a3bbf7252dfa1813 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.108811 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:37.133291 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_5d473cabff2e46e0b531fbce28d9d558 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 1 got 0
I20250112 02:52:37.133643 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:37.133908 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:37.134557 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:37.137406 855 leader_election.cc:304] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
W20250112 02:52:37.137686 855 raft_consensus.cc:2681] T d659c5a4f7a446c6aaf2283f65b45106 P fe36951a83fb4e57b26e187edbcbfe3d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 02:52:37.138890 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 1 got 0
I20250112 02:52:37.139266 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:37.140926 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:37.141727 910 raft_consensus.cc:2463] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:37.143672 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:37.143872 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3f5b418c17084420b867498fda79fb0a (table default.Table_96d1a09a902340ff8e05efb75faf3854 [id=ad0be37e6b6f4b9597dcb4cbd50d5b12]). Expected version 3 got 2
I20250112 02:52:37.144080 2036 ts_tablet_manager.cc:1918] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.144089 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:37.144174 2036 log.cc:1198] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/02d0f67141dc46b6a87ca0ad080d5bfe
I20250112 02:52:37.144465 2036 ts_tablet_manager.cc:1939] T 02d0f67141dc46b6a87ca0ad080d5bfe P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.144716 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 65) took 2896 ms. Trace:
I20250112 02:52:37.144784 2036 rpcz_store.cc:276] 0112 02:52:34.248130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.278354 (+ 30224us) service_pool.cc:225] Handling call
0112 02:52:37.102221 (+2823867us) ts_tablet_manager.cc:1145] Deleting tablet 02d0f67141dc46b6a87ca0ad080d5bfe
0112 02:52:37.102229 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.141626 (+ 39397us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.144001 (+ 2375us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.144705 (+ 704us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2823727}
I20250112 02:52:37.144910 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:37.145044 2036 tablet_replica.cc:331] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.145337 2036 raft_consensus.cc:2238] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.145458 2036 raft_consensus.cc:2267] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.145566 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
I20250112 02:52:37.145749 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:37.146004 2036 ts_tablet_manager.cc:1905] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.146409 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:37.147166 2003 tablet_replica.cc:331] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:37.147411 2003 raft_consensus.cc:2238] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.147605 2003 raft_consensus.cc:2267] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.148478 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5cc414d6420c41839621ab858067548b (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]). Expected version 1 got 0
I20250112 02:52:37.154218 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet c4a3486ef011492c8a1df8d6278adedf (table default.Table_33b6530b130041f0a2a4a7240e92ff15 [id=1d614fd3947a43bcac5a3d40cad6d80b]) successfully deleted
I20250112 02:52:37.154412 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet cc62dcbf67e84e36ae1fbe127ccd4879 (table default.Table_fd0169db0f984d3cbc0d8624dda686bc [id=9e09b6670c6e4e57b62f31fd4755c18b]) successfully deleted
I20250112 02:52:37.154533 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet d659c5a4f7a446c6aaf2283f65b45106 (table default.Table_218fd160176b4f55937d334142f2ecbd [id=990e8109058d4dffbbd0ae23fc43783b]) successfully deleted
I20250112 02:52:37.154662 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 02d0f67141dc46b6a87ca0ad080d5bfe (table default.Table_3a84890c771c4574a25d33543fe7217d [id=49a61d0f688b488da9a139f0a64d2834]) successfully deleted
I20250112 02:52:37.155802 2003 ts_tablet_manager.cc:1905] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.156131 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:37.156436 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:37.157282 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 48634d6d78784fae98512b242e5476df (table default.Table_7cbe570a89784c18b4a14a8242062071 [id=fee2f9e65e314ce9a1fc1fb6dfcde317]). Expected version 1 got 0
I20250112 02:52:37.157585 1177 tablet_bootstrap.cc:492] T b3c5a861a90840d0a33b780ec75bffd2 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:37.158311 1181 tablet_bootstrap.cc:492] T a437952889494b0e820e41fdfffe708d P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:37.158421 1181 ts_tablet_manager.cc:1397] T a437952889494b0e820e41fdfffe708d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.010s
I20250112 02:52:37.159067 1181 raft_consensus.cc:357] T a437952889494b0e820e41fdfffe708d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.159241 1181 raft_consensus.cc:383] T a437952889494b0e820e41fdfffe708d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.159307 1181 raft_consensus.cc:738] T a437952889494b0e820e41fdfffe708d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.159502 1181 consensus_queue.cc:260] T a437952889494b0e820e41fdfffe708d P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.160168 1181 ts_tablet_manager.cc:1428] T a437952889494b0e820e41fdfffe708d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:37.165177 1181 tablet_bootstrap.cc:492] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:37.166998 1181 tablet_bootstrap.cc:654] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.167286 2036 ts_tablet_manager.cc:1918] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.167404 2036 log.cc:1198] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/1a8a178bafcb457f9b0e9587df560493
I20250112 02:52:37.173952 1177 ts_tablet_manager.cc:1397] T b3c5a861a90840d0a33b780ec75bffd2 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.102s user 0.004s sys 0.006s
I20250112 02:52:37.157799 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet f8c01b80fc8242d98fdfbf767212144d (table default.Table_57b3861e9f704dc09d466241c1926fc1 [id=9807191b6e5c4d478cd47a54464fb9ce]). Expected version 2 got 1
I20250112 02:52:37.175736 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet f31b777e7e0f43a28839d8c48c6a6590 (table default.Table_5fb592e41cc4406aa27546e7b41fdb88 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 1 got 0
I20250112 02:52:37.176546 2036 ts_tablet_manager.cc:1939] T 1a8a178bafcb457f9b0e9587df560493 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.177029 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:37.177312 1177 raft_consensus.cc:357] T b3c5a861a90840d0a33b780ec75bffd2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.177418 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 66) took 2929 ms. Trace:
I20250112 02:52:37.177485 2036 rpcz_store.cc:276] 0112 02:52:34.248290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.278506 (+ 30216us) service_pool.cc:225] Handling call
0112 02:52:37.145008 (+2866502us) ts_tablet_manager.cc:1145] Deleting tablet 1a8a178bafcb457f9b0e9587df560493
0112 02:52:37.145015 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.148395 (+ 3380us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.167207 (+ 18812us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.177404 (+ 10197us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2866413}
I20250112 02:52:37.177752 2036 tablet_replica.cc:331] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.177879 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 1a8a178bafcb457f9b0e9587df560493 (table default.Table_3c1f0242d17443b5a758967cf82ca0ca [id=02303baf5298474488dbf02c831f3d4d]) successfully deleted
I20250112 02:52:37.177912 2036 raft_consensus.cc:2238] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.178014 2036 raft_consensus.cc:2267] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.178409 2036 ts_tablet_manager.cc:1905] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.178556 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:37.179432 2024 log.cc:1198] T 64707ee432fb4e96a3bbf7252dfa1813 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/64707ee432fb4e96a3bbf7252dfa1813
I20250112 02:52:37.145445 2228 consensus_queue.cc:1035] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:37.179577 1177 raft_consensus.cc:383] T b3c5a861a90840d0a33b780ec75bffd2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.179636 1177 raft_consensus.cc:738] T b3c5a861a90840d0a33b780ec75bffd2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.179867 2024 ts_tablet_manager.cc:1939] T 64707ee432fb4e96a3bbf7252dfa1813 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:37.157189 1173 tablet_bootstrap.cc:492] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:37.179843 1177 consensus_queue.cc:260] T b3c5a861a90840d0a33b780ec75bffd2 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.179989 1173 ts_tablet_manager.cc:1397] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.007s
I20250112 02:52:37.180121 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 58) took 2700 ms. Trace:
I20250112 02:52:37.180191 2024 rpcz_store.cc:276] 0112 02:52:34.479344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.479595 (+ 251us) service_pool.cc:225] Handling call
0112 02:52:37.123231 (+2643636us) ts_tablet_manager.cc:1145] Deleting tablet 64707ee432fb4e96a3bbf7252dfa1813
0112 02:52:37.123241 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.130136 (+ 6895us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.132740 (+ 2604us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.180109 (+ 47369us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2643450}
I20250112 02:52:37.180332 1177 ts_tablet_manager.cc:1428] T b3c5a861a90840d0a33b780ec75bffd2 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.006s user 0.000s sys 0.000s
I20250112 02:52:37.180553 1177 tablet_bootstrap.cc:492] T b9537fcd96eb41cc8f680075ec71137b P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:37.180617 1173 raft_consensus.cc:357] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.180757 1173 raft_consensus.cc:383] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.180805 1173 raft_consensus.cc:738] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.180963 1173 consensus_queue.cc:260] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.181423 1173 ts_tablet_manager.cc:1428] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 02:52:37.182806 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 64707ee432fb4e96a3bbf7252dfa1813 (table default.Table_c26869f6cbd540fe980dcf207b5c729d [id=cdd6a2097792484d84bca856f30d7ac1]) successfully deleted
I20250112 02:52:37.185472 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:37.188319 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:37.191617 2024 tablet_replica.cc:331] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:37.191881 2003 ts_tablet_manager.cc:1918] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.191982 2003 log.cc:1198] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/9163878cfeb64693a9c5493e812546af
I20250112 02:52:37.192458 2179 raft_consensus.cc:491] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.192559 2179 raft_consensus.cc:513] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.193046 2179 leader_election.cc:290] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:37.193760 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12c5d58e6c246c983e8a533906ce77d" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:37.193843 713 leader_election.cc:343] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: a12c5d58e6c246c983e8a533906ce77d
I20250112 02:52:37.194643 2003 ts_tablet_manager.cc:1939] T 9163878cfeb64693a9c5493e812546af P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
W20250112 02:52:37.195485 714 leader_election.cc:343] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.195580 714 leader_election.cc:304] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:37.195921 2179 raft_consensus.cc:2743] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:37.196024 1177 tablet_bootstrap.cc:654] T b9537fcd96eb41cc8f680075ec71137b P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.197863 1173 tablet_bootstrap.cc:492] T b9d525dae23744a4adfc2f58cb9945ce P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:37.199290 1173 tablet_bootstrap.cc:654] T b9d525dae23744a4adfc2f58cb9945ce P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.209199 2024 raft_consensus.cc:2238] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.214133 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 72) took 2966 ms. Trace:
I20250112 02:52:37.214227 2003 rpcz_store.cc:276] 0112 02:52:34.247594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.247636 (+ 42us) service_pool.cc:225] Handling call
0112 02:52:37.147122 (+2899486us) ts_tablet_manager.cc:1145] Deleting tablet 9163878cfeb64693a9c5493e812546af
0112 02:52:37.147133 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.160611 (+ 13478us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.191801 (+ 31190us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.214112 (+ 22311us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2899299}
I20250112 02:52:37.214390 2024 raft_consensus.cc:2267] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.215862 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 9163878cfeb64693a9c5493e812546af (table default.Table_426c5d1d19a94f0d8110a9cbee9a9044 [id=69f55e532f714a2d84790df8ff19e6dc]) successfully deleted
I20250112 02:52:37.216512 2024 ts_tablet_manager.cc:1905] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.217684 2003 tablet_replica.cc:331] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:37.217895 2003 raft_consensus.cc:2238] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.218084 2003 raft_consensus.cc:2267] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.218783 742 tablet_service.cc:1467] Processing CreateTablet for tablet b114b98d1cf4419a8bfeef558d818c01 (DEFAULT_TABLE table=default.Table_ac83a75456fe4f0ea7f10cb28e29cd28 [id=90990933135040478da455176db75546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:37.219094 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b114b98d1cf4419a8bfeef558d818c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:37.221838 2003 ts_tablet_manager.cc:1905] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.222411 1034 tablet_service.cc:1467] Processing CreateTablet for tablet b114b98d1cf4419a8bfeef558d818c01 (DEFAULT_TABLE table=default.Table_ac83a75456fe4f0ea7f10cb28e29cd28 [id=90990933135040478da455176db75546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:37.222692 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b114b98d1cf4419a8bfeef558d818c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:37.226447 889 tablet_service.cc:1467] Processing CreateTablet for tablet b114b98d1cf4419a8bfeef558d818c01 (DEFAULT_TABLE table=default.Table_ac83a75456fe4f0ea7f10cb28e29cd28 [id=90990933135040478da455176db75546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:37.226801 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b114b98d1cf4419a8bfeef558d818c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:37.231868 742 tablet_service.cc:1467] Processing CreateTablet for tablet d86417fc1c844cdf85323b94c6dce6d5 (DEFAULT_TABLE table=default.Table_a39e287f411145fca081afaaf41d3311 [id=99414b4a995a4697ae1022a60cf2a359]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:37.232252 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d86417fc1c844cdf85323b94c6dce6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:37.232836 1181 tablet_bootstrap.cc:492] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:37.232955 1181 ts_tablet_manager.cc:1397] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.068s user 0.005s sys 0.007s
I20250112 02:52:37.233613 1181 raft_consensus.cc:357] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.233769 1181 raft_consensus.cc:383] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.233825 1181 raft_consensus.cc:738] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.234009 1181 consensus_queue.cc:260] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.234155 2117 consensus_queue.cc:1035] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:37.234546 1181 ts_tablet_manager.cc:1428] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:37.234776 1181 tablet_bootstrap.cc:492] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:37.240604 2047 raft_consensus.cc:491] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.240716 2047 raft_consensus.cc:513] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.240782 891 tablet_service.cc:1467] Processing CreateTablet for tablet 681a930541a84f0197ecafe670278e70 (DEFAULT_TABLE table=default.Table_8293882503214cf284e12ff8cc75b9c7 [id=15f8694f2e1a403fbf934065a1a79e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:37.241052 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 681a930541a84f0197ecafe670278e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:37.240972 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 681a930541a84f0197ecafe670278e70 (DEFAULT_TABLE table=default.Table_8293882503214cf284e12ff8cc75b9c7 [id=15f8694f2e1a403fbf934065a1a79e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:37.241317 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 681a930541a84f0197ecafe670278e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:37.241313 2047 leader_election.cc:290] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:37.241617 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 4cfe1a1595014c27bbd350cec169d71a (DEFAULT_TABLE table=default.Table_93133ac10ebe44d58e69c21d228c1c5e [id=d8ecda6d109a47c494582f7a77cd92f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:37.241674 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb76746b3a846c3aba531c04bf63966" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:37.241887 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cfe1a1595014c27bbd350cec169d71a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:37.242071 714 leader_election.cc:343] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.243844 2003 ts_tablet_manager.cc:1918] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.243925 2003 log.cc:1198] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/5dd48d0e68014e1aa0c44620527baec9
I20250112 02:52:37.244166 2003 ts_tablet_manager.cc:1939] T 5dd48d0e68014e1aa0c44620527baec9 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:37.244895 756 tablet_service.cc:1467] Processing CreateTablet for tablet 681a930541a84f0197ecafe670278e70 (DEFAULT_TABLE table=default.Table_8293882503214cf284e12ff8cc75b9c7 [id=15f8694f2e1a403fbf934065a1a79e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:37.245162 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 681a930541a84f0197ecafe670278e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:37.245945 1181 tablet_bootstrap.cc:654] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.247242 889 tablet_service.cc:1467] Processing CreateTablet for tablet d86417fc1c844cdf85323b94c6dce6d5 (DEFAULT_TABLE table=default.Table_a39e287f411145fca081afaaf41d3311 [id=99414b4a995a4697ae1022a60cf2a359]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:37.245107 755 tablet_service.cc:1467] Processing CreateTablet for tablet 4cfe1a1595014c27bbd350cec169d71a (DEFAULT_TABLE table=default.Table_93133ac10ebe44d58e69c21d228c1c5e [id=d8ecda6d109a47c494582f7a77cd92f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:37.247606 2047 raft_consensus.cc:491] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.247736 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cfe1a1595014c27bbd350cec169d71a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:37.247700 2047 raft_consensus.cc:513] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.247941 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d86417fc1c844cdf85323b94c6dce6d5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:37.248234 713 leader_election.cc:343] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 6cb76746b3a846c3aba531c04bf63966
I20250112 02:52:37.248334 713 leader_election.cc:304] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:37.249051 2047 leader_election.cc:290] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:37.249053 2179 raft_consensus.cc:2743] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:37.249374 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe614eaeb793473d9b2da206b7345562" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:37.249739 714 leader_election.cc:343] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.251991 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe614eaeb793473d9b2da206b7345562" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
W20250112 02:52:37.252373 713 leader_election.cc:343] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Illegal state: Tablet not RUNNING: STOPPED
I20250112 02:52:37.252467 713 leader_election.cc:304] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:37.252655 2179 raft_consensus.cc:2743] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:37.255103 1034 tablet_service.cc:1467] Processing CreateTablet for tablet d86417fc1c844cdf85323b94c6dce6d5 (DEFAULT_TABLE table=default.Table_a39e287f411145fca081afaaf41d3311 [id=99414b4a995a4697ae1022a60cf2a359]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:37.255713 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d86417fc1c844cdf85323b94c6dce6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:37.244434 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 73) took 2996 ms. Trace:
I20250112 02:52:37.257829 2003 rpcz_store.cc:276] 0112 02:52:34.248005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.248046 (+ 41us) service_pool.cc:225] Handling call
0112 02:52:37.217636 (+2969590us) ts_tablet_manager.cc:1145] Deleting tablet 5dd48d0e68014e1aa0c44620527baec9
0112 02:52:37.217646 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.226767 (+ 9121us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.243781 (+ 17014us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.244423 (+ 642us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2969397}
I20250112 02:52:37.258152 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 5dd48d0e68014e1aa0c44620527baec9 (table default.Table_66559fa571be4748b005f229c7089c09 [id=5431f5161ddf40d083b94b5fcb6b7561]) successfully deleted
I20250112 02:52:37.258263 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 74) took 3009 ms. Trace:
I20250112 02:52:37.258323 2003 rpcz_store.cc:276] 0112 02:52:34.248529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.248574 (+ 45us) service_pool.cc:225] Handling call
0112 02:52:37.258162 (+3009588us) ts_tablet_manager.cc:1145] Deleting tablet e5c10fad790046a88fbc8ebe0303c481
0112 02:52:37.258172 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.258249 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3009380}
I20250112 02:52:37.258692 2003 tablet_replica.cc:331] T 9d5b3df88a0c45dba879b3d506710614 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
W20250112 02:52:37.258730 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet e5c10fad790046a88fbc8ebe0303c481 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5c10fad790046a88fbc8ebe0303c481 already in progress: creating tablet
I20250112 02:52:37.258841 2003 raft_consensus.cc:2238] T 9d5b3df88a0c45dba879b3d506710614 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.258953 2003 raft_consensus.cc:2267] T 9d5b3df88a0c45dba879b3d506710614 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.259689 889 tablet_service.cc:1467] Processing CreateTablet for tablet 4cfe1a1595014c27bbd350cec169d71a (DEFAULT_TABLE table=default.Table_93133ac10ebe44d58e69c21d228c1c5e [id=d8ecda6d109a47c494582f7a77cd92f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:37.260178 2003 ts_tablet_manager.cc:1905] T 9d5b3df88a0c45dba879b3d506710614 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.260761 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cfe1a1595014c27bbd350cec169d71a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:37.260550 2036 ts_tablet_manager.cc:1918] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.263487 2036 log.cc:1198] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/fe614eaeb793473d9b2da206b7345562
I20250112 02:52:37.265769 2399 raft_consensus.cc:491] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.267590 2399 raft_consensus.cc:513] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.267765 2036 ts_tablet_manager.cc:1939] T fe614eaeb793473d9b2da206b7345562 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.268000 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 67) took 3019 ms. Trace:
I20250112 02:52:37.268071 2036 rpcz_store.cc:276] 0112 02:52:34.248400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.278598 (+ 30198us) service_pool.cc:225] Handling call
0112 02:52:37.177714 (+2899116us) ts_tablet_manager.cc:1145] Deleting tablet fe614eaeb793473d9b2da206b7345562
0112 02:52:37.177723 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.211652 (+ 33929us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.260481 (+ 48829us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.267987 (+ 7506us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2899026}
I20250112 02:52:37.268175 2399 leader_election.cc:290] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:37.268326 2036 tablet_replica.cc:331] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.268492 2036 raft_consensus.cc:2238] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.268832 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af55dab202444358ea1a0674d37946d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:37.269731 2024 ts_tablet_manager.cc:1918] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.269697 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af55dab202444358ea1a0674d37946d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:37.270213 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet fe614eaeb793473d9b2da206b7345562 (table default.Table_b2b3c738a24f451eac39fb1a6399f37a [id=7c6c2713869e476cb23d157a285a5afd]) successfully deleted
W20250112 02:52:37.270612 854 leader_election.cc:343] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.271828 2036 raft_consensus.cc:2267] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:37.272300 855 leader_election.cc:343] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.272374 2036 ts_tablet_manager.cc:1905] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.272408 855 leader_election.cc:304] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:37.273101 2399 raft_consensus.cc:2743] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:37.275592 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4d25815a5df94e099e7481c48c3da793 consensus state has no leader
I20250112 02:52:37.275795 2024 log.cc:1198] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/3fd498ccce484a89855c982762086890
I20250112 02:52:37.273445 2215 consensus_queue.cc:1035] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:37.272285 1173 tablet_bootstrap.cc:492] T b9d525dae23744a4adfc2f58cb9945ce P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:37.276362 1173 ts_tablet_manager.cc:1397] T b9d525dae23744a4adfc2f58cb9945ce P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.079s user 0.000s sys 0.011s
I20250112 02:52:37.276759 2399 raft_consensus.cc:491] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.276862 2399 raft_consensus.cc:513] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.277104 1173 raft_consensus.cc:357] T b9d525dae23744a4adfc2f58cb9945ce P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.277243 1173 raft_consensus.cc:383] T b9d525dae23744a4adfc2f58cb9945ce P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.277289 1173 raft_consensus.cc:738] T b9d525dae23744a4adfc2f58cb9945ce P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.277403 2399 leader_election.cc:290] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:37.277711 2024 ts_tablet_manager.cc:1939] T 3fd498ccce484a89855c982762086890 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:37.278388 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 59) took 2797 ms. Trace:
I20250112 02:52:37.278501 2024 rpcz_store.cc:276] 0112 02:52:34.480792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.491382 (+ 10590us) service_pool.cc:225] Handling call
0112 02:52:37.180391 (+2689009us) ts_tablet_manager.cc:1145] Deleting tablet 3fd498ccce484a89855c982762086890
0112 02:52:37.180398 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.224328 (+ 43930us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.269649 (+ 45321us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.278373 (+ 8724us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2688789}
I20250112 02:52:37.278718 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd6d31155034908ac44a90c3a2ea7d6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:37.278976 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 3fd498ccce484a89855c982762086890 (table default.Table_945067c243f84394957a328d65592441 [id=5a5f3290dab54cf0b13c88efd52c90c5]) successfully deleted
I20250112 02:52:37.279261 2024 tablet_replica.cc:331] T 015412a5fa304453bd8d1be1c3cd7a46 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:37.279428 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd6d31155034908ac44a90c3a2ea7d6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:37.279496 2024 raft_consensus.cc:2238] T 015412a5fa304453bd8d1be1c3cd7a46 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.279633 2024 raft_consensus.cc:2267] T 015412a5fa304453bd8d1be1c3cd7a46 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:37.279863 854 leader_election.cc:343] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:37.279965 855 leader_election.cc:343] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.280066 855 leader_election.cc:304] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:37.280175 2024 ts_tablet_manager.cc:1905] T 015412a5fa304453bd8d1be1c3cd7a46 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.280654 2117 raft_consensus.cc:2743] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:37.281224 1101 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:37.283640 1173 consensus_queue.cc:260] T b9d525dae23744a4adfc2f58cb9945ce P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.284274 2222 consensus_queue.cc:1035] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:37.292011 1173 ts_tablet_manager.cc:1428] T b9d525dae23744a4adfc2f58cb9945ce P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250112 02:52:37.292308 1173 tablet_bootstrap.cc:492] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:37.293216 1181 tablet_bootstrap.cc:492] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:37.293346 1181 ts_tablet_manager.cc:1397] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.002s
I20250112 02:52:37.293969 1181 raft_consensus.cc:357] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.294132 1181 raft_consensus.cc:383] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.294184 1181 raft_consensus.cc:738] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.294358 1173 tablet_bootstrap.cc:654] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.294420 1181 consensus_queue.cc:260] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.294900 1181 ts_tablet_manager.cc:1428] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:37.295125 1181 tablet_bootstrap.cc:492] T 8c821ea3ca4946c290c17135366e0cac P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:37.296455 1181 tablet_bootstrap.cc:654] T 8c821ea3ca4946c290c17135366e0cac P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.297221 2036 ts_tablet_manager.cc:1918] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.297560 2036 log.cc:1198] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/41e166b3f8414ababbd4a3f07fb90c4c
I20250112 02:52:37.298213 2024 ts_tablet_manager.cc:1918] T 015412a5fa304453bd8d1be1c3cd7a46 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.298290 2024 log.cc:1198] T 015412a5fa304453bd8d1be1c3cd7a46 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/015412a5fa304453bd8d1be1c3cd7a46
I20250112 02:52:37.298565 2024 ts_tablet_manager.cc:1939] T 015412a5fa304453bd8d1be1c3cd7a46 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:37.298681 2036 ts_tablet_manager.cc:1939] T 41e166b3f8414ababbd4a3f07fb90c4c P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.298789 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 60) took 2807 ms. Trace:
I20250112 02:52:37.298851 2024 rpcz_store.cc:276] 0112 02:52:34.491774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.491813 (+ 39us) service_pool.cc:225] Handling call
0112 02:52:37.279223 (+2787410us) ts_tablet_manager.cc:1145] Deleting tablet 015412a5fa304453bd8d1be1c3cd7a46
0112 02:52:37.279231 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.282981 (+ 3750us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.298145 (+ 15164us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.298777 (+ 632us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2787255}
I20250112 02:52:37.299206 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 015412a5fa304453bd8d1be1c3cd7a46 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]) successfully deleted
I20250112 02:52:37.299651 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 68) took 3051 ms. Trace:
I20250112 02:52:37.299903 2036 rpcz_store.cc:276] 0112 02:52:34.248484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.278692 (+ 30208us) service_pool.cc:225] Handling call
0112 02:52:37.268283 (+2989591us) ts_tablet_manager.cc:1145] Deleting tablet 41e166b3f8414ababbd4a3f07fb90c4c
0112 02:52:37.268292 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.274921 (+ 6629us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.297057 (+ 22136us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.299527 (+ 2470us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2989480}
I20250112 02:52:37.300544 2024 tablet_replica.cc:331] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:37.300654 2036 tablet_replica.cc:331] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.300714 2024 raft_consensus.cc:2238] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:37.300891 2024 pending_rounds.cc:62] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6: Trying to abort 1 pending ops.
I20250112 02:52:37.300909 2036 raft_consensus.cc:2238] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.300931 2024 pending_rounds.cc:69] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319862696144896 op_type: NO_OP noop_request { }
I20250112 02:52:37.301008 2024 raft_consensus.cc:2883] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:37.301083 2024 raft_consensus.cc:2267] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.301319 2036 raft_consensus.cc:2267] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.301702 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 41e166b3f8414ababbd4a3f07fb90c4c (table default.Table_11b4065cfb0449b68950d63e65f566a2 [id=0e325561bbdb468e8a64ca9808e2f3cf]) successfully deleted
I20250112 02:52:37.302181 2036 ts_tablet_manager.cc:1905] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.306428 2222 raft_consensus.cc:491] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.306582 2222 raft_consensus.cc:513] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.307228 2222 leader_election.cc:290] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:37.307734 2024 ts_tablet_manager.cc:1905] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.308763 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06edfca60eed4dd3a2dee7b662b0970c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:37.309223 855 leader_election.cc:343] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.308120 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06edfca60eed4dd3a2dee7b662b0970c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:37.311000 854 leader_election.cc:343] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.311094 854 leader_election.cc:304] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:37.311525 2222 raft_consensus.cc:2743] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:37.312680 2036 ts_tablet_manager.cc:1918] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.312816 2036 log.cc:1198] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/04f135b226774285a077e5806131d91b
I20250112 02:52:37.313295 2036 ts_tablet_manager.cc:1939] T 04f135b226774285a077e5806131d91b P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.313704 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 69) took 3065 ms. Trace:
I20250112 02:52:37.313796 2036 rpcz_store.cc:276] 0112 02:52:34.248565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.278814 (+ 30249us) service_pool.cc:225] Handling call
0112 02:52:37.300588 (+3021774us) ts_tablet_manager.cc:1145] Deleting tablet 04f135b226774285a077e5806131d91b
0112 02:52:37.300598 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.306959 (+ 6361us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.312559 (+ 5600us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.313669 (+ 1110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3021646}
I20250112 02:52:37.314239 2036 tablet_replica.cc:331] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.314313 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 04f135b226774285a077e5806131d91b (table default.Table_d1e5cef5fa3e4d90bada176f80bd4180 [id=f93b5bb22ec1424ea047383ed9713618]) successfully deleted
I20250112 02:52:37.314553 2003 ts_tablet_manager.cc:1918] T 9d5b3df88a0c45dba879b3d506710614 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.314536 2036 raft_consensus.cc:2238] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.314656 2003 log.cc:1198] T 9d5b3df88a0c45dba879b3d506710614 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/9d5b3df88a0c45dba879b3d506710614
I20250112 02:52:37.314976 2036 raft_consensus.cc:2267] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.315716 2036 ts_tablet_manager.cc:1905] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.317827 1181 tablet_bootstrap.cc:492] T 8c821ea3ca4946c290c17135366e0cac P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:37.317953 1181 ts_tablet_manager.cc:1397] T 8c821ea3ca4946c290c17135366e0cac P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20250112 02:52:37.318631 1181 raft_consensus.cc:357] T 8c821ea3ca4946c290c17135366e0cac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.318821 1181 raft_consensus.cc:383] T 8c821ea3ca4946c290c17135366e0cac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.318881 1181 raft_consensus.cc:738] T 8c821ea3ca4946c290c17135366e0cac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.319114 1181 consensus_queue.cc:260] T 8c821ea3ca4946c290c17135366e0cac P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.319664 1181 ts_tablet_manager.cc:1428] T 8c821ea3ca4946c290c17135366e0cac P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:37.319905 1181 tablet_bootstrap.cc:492] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:37.320776 2003 ts_tablet_manager.cc:1939] T 9d5b3df88a0c45dba879b3d506710614 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:37.321605 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 75) took 3072 ms. Trace:
I20250112 02:52:37.321693 2003 rpcz_store.cc:276] 0112 02:52:34.248980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.249022 (+ 42us) service_pool.cc:225] Handling call
0112 02:52:37.258658 (+3009636us) ts_tablet_manager.cc:1145] Deleting tablet 9d5b3df88a0c45dba879b3d506710614
0112 02:52:37.258665 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.263023 (+ 4358us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.314470 (+ 51447us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.321587 (+ 7117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3009468}
I20250112 02:52:37.322077 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 76) took 3072 ms. Trace:
I20250112 02:52:37.322085 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 9d5b3df88a0c45dba879b3d506710614 (table default.Table_49b5979e0b174043b61f2f1d61a5b58c [id=b81cf271b11e45dcb0b6bce102cd7a0c]) successfully deleted
I20250112 02:52:37.322152 2003 rpcz_store.cc:276] 0112 02:52:34.249356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.249396 (+ 40us) service_pool.cc:225] Handling call
0112 02:52:37.321973 (+3072577us) ts_tablet_manager.cc:1145] Deleting tablet 7a34d03a80ae402db6607261a5c40b60
0112 02:52:37.321983 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.322064 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3072409}
I20250112 02:52:37.322434 2003 tablet_replica.cc:331] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
W20250112 02:52:37.322566 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 7a34d03a80ae402db6607261a5c40b60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a34d03a80ae402db6607261a5c40b60 already in progress: creating tablet
I20250112 02:52:37.322628 2003 raft_consensus.cc:2238] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:37.322923 2003 raft_consensus.cc:2267] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.323454 2003 ts_tablet_manager.cc:1905] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.328312 2036 ts_tablet_manager.cc:1918] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.328449 1181 tablet_bootstrap.cc:654] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.328519 2036 log.cc:1198] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/1a1e3af2883341e9b957f6ed748e604d
I20250112 02:52:37.329322 2036 ts_tablet_manager.cc:1939] T 1a1e3af2883341e9b957f6ed748e604d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.329815 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 70) took 3081 ms. Trace:
I20250112 02:52:37.329946 2036 rpcz_store.cc:276] 0112 02:52:34.248664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.278949 (+ 30285us) service_pool.cc:225] Handling call
0112 02:52:37.314170 (+3035221us) ts_tablet_manager.cc:1145] Deleting tablet 1a1e3af2883341e9b957f6ed748e604d
0112 02:52:37.314180 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.322880 (+ 8700us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.328196 (+ 5316us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.329798 (+ 1602us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3035095}
I20250112 02:52:37.330436 2036 tablet_replica.cc:331] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.330538 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 1a1e3af2883341e9b957f6ed748e604d (table default.Table_b3ed6c7a79ed43568b8743f8aea9f67e [id=902e40c3f1e9401e934a4a2652f3dc22]) successfully deleted
I20250112 02:52:37.330713 2036 raft_consensus.cc:2238] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.331113 2036 raft_consensus.cc:2267] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.331964 2036 ts_tablet_manager.cc:1905] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.336442 2024 ts_tablet_manager.cc:1918] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.339391 2024 log.cc:1198] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/f7cd0927c6a4401cbe69f55d4db9963a
I20250112 02:52:37.340088 2024 ts_tablet_manager.cc:1939] T f7cd0927c6a4401cbe69f55d4db9963a P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:37.341133 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 61) took 2849 ms. Trace:
I20250112 02:52:37.341356 1177 tablet_bootstrap.cc:492] T b9537fcd96eb41cc8f680075ec71137b P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:37.341490 1177 ts_tablet_manager.cc:1397] T b9537fcd96eb41cc8f680075ec71137b P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.161s user 0.004s sys 0.000s
I20250112 02:52:37.342229 1177 raft_consensus.cc:357] T b9537fcd96eb41cc8f680075ec71137b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.342406 1177 raft_consensus.cc:383] T b9537fcd96eb41cc8f680075ec71137b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.342497 1177 raft_consensus.cc:738] T b9537fcd96eb41cc8f680075ec71137b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.342542 2024 rpcz_store.cc:276] 0112 02:52:34.492085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.492122 (+ 37us) service_pool.cc:225] Handling call
0112 02:52:37.300499 (+2808377us) ts_tablet_manager.cc:1145] Deleting tablet f7cd0927c6a4401cbe69f55d4db9963a
0112 02:52:37.300507 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.319032 (+ 18525us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.336370 (+ 17338us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.341117 (+ 4747us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2808216}
I20250112 02:52:37.343353 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet f7cd0927c6a4401cbe69f55d4db9963a (table default.Table_8b4baeb21bec40af9997451f3eceddf1 [id=b7a72bcbde114d5992782721007a0cd4]) successfully deleted
I20250112 02:52:37.343595 2036 ts_tablet_manager.cc:1918] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.343721 2036 log.cc:1198] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/49bb6d20cfa0453e90e048825114525e
I20250112 02:52:37.343936 2024 tablet_replica.cc:331] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:37.344236 2036 ts_tablet_manager.cc:1939] T 49bb6d20cfa0453e90e048825114525e P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.345137 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 71) took 3096 ms. Trace:
I20250112 02:52:37.345247 2036 rpcz_store.cc:276] 0112 02:52:34.248764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.279063 (+ 30299us) service_pool.cc:225] Handling call
0112 02:52:37.330373 (+3051310us) ts_tablet_manager.cc:1145] Deleting tablet 49bb6d20cfa0453e90e048825114525e
0112 02:52:37.330382 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.336025 (+ 5643us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.343463 (+ 7438us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.345121 (+ 1658us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":135,"tablet-delete.queue_time_us":3051176}
I20250112 02:52:37.345697 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 49bb6d20cfa0453e90e048825114525e (table default.Table_f359341b73c14c60b859f06dc5f428a7 [id=bedc9786ee5b40a9bdcbd3cd0be38361]) successfully deleted
I20250112 02:52:37.345744 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 21) took 3262 ms. Trace:
I20250112 02:52:37.345842 2036 rpcz_store.cc:276] 0112 02:52:34.083316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.083492 (+ 176us) service_pool.cc:225] Handling call
0112 02:52:37.345579 (+3262087us) ts_tablet_manager.cc:1145] Deleting tablet bd5b3c712c0f448ea9b14ab93dd731ab
0112 02:52:37.345586 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.345711 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3062322}
W20250112 02:52:37.346799 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet bd5b3c712c0f448ea9b14ab93dd731ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd5b3c712c0f448ea9b14ab93dd731ab already in progress: creating tablet
I20250112 02:52:37.347023 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 22) took 3262 ms. Trace:
W20250112 02:52:37.347573 991 consensus_peers.cc:487] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb0aba0139cf479b80b32f3dfc936535. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:37.347146 2036 rpcz_store.cc:276] 0112 02:52:34.084206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.084249 (+ 43us) service_pool.cc:225] Handling call
0112 02:52:37.346875 (+3262626us) ts_tablet_manager.cc:1145] Deleting tablet b1d9fae290ec4281839991f9d40b105c
0112 02:52:37.346884 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.347006 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3063538}
W20250112 02:52:37.348305 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet b1d9fae290ec4281839991f9d40b105c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1d9fae290ec4281839991f9d40b105c already in progress: creating tablet
I20250112 02:52:37.348830 2036 tablet_replica.cc:331] T 8c821ea3ca4946c290c17135366e0cac P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.349071 2036 raft_consensus.cc:2238] T 8c821ea3ca4946c290c17135366e0cac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.349267 2036 raft_consensus.cc:2267] T 8c821ea3ca4946c290c17135366e0cac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.356676 2222 raft_consensus.cc:491] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.356786 2222 raft_consensus.cc:513] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.357359 2222 leader_election.cc:290] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:37.357702 2036 ts_tablet_manager.cc:1905] T 8c821ea3ca4946c290c17135366e0cac P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.357864 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777272e7159b438ebff4db93698411d0" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:37.358261 854 leader_election.cc:343] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.358527 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777272e7159b438ebff4db93698411d0" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:37.358960 855 leader_election.cc:343] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.359057 855 leader_election.cc:304] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:37.359413 2222 raft_consensus.cc:2743] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:37.361210 2024 raft_consensus.cc:2238] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.363590 2024 raft_consensus.cc:2267] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.364388 2024 ts_tablet_manager.cc:1905] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.366823 2222 raft_consensus.cc:491] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.366930 2222 raft_consensus.cc:513] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.367465 2222 leader_election.cc:290] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:37.367925 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9537fcd96eb41cc8f680075ec71137b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:37.368541 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9537fcd96eb41cc8f680075ec71137b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:37.368700 1049 raft_consensus.cc:2463] T b9537fcd96eb41cc8f680075ec71137b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:37.369002 854 leader_election.cc:304] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:37.369221 2222 raft_consensus.cc:2798] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:37.369298 2222 raft_consensus.cc:491] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:37.369364 2222 raft_consensus.cc:3054] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.369762 1173 tablet_bootstrap.cc:492] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:37.369916 1173 ts_tablet_manager.cc:1397] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.078s user 0.009s sys 0.005s
I20250112 02:52:37.370075 2036 ts_tablet_manager.cc:1918] T 8c821ea3ca4946c290c17135366e0cac P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.370450 2036 log.cc:1198] T 8c821ea3ca4946c290c17135366e0cac P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/8c821ea3ca4946c290c17135366e0cac
I20250112 02:52:37.370725 1173 raft_consensus.cc:357] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.370914 1173 raft_consensus.cc:383] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.370987 1173 raft_consensus.cc:738] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.371181 1173 consensus_queue.cc:260] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.371510 2036 ts_tablet_manager.cc:1939] T 8c821ea3ca4946c290c17135366e0cac P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.371649 2222 raft_consensus.cc:513] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.372470 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9537fcd96eb41cc8f680075ec71137b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:37.372617 1049 raft_consensus.cc:3054] T b9537fcd96eb41cc8f680075ec71137b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.372834 2222 leader_election.cc:290] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:37.373068 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9537fcd96eb41cc8f680075ec71137b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:37.373828 1177 consensus_queue.cc:260] T b9537fcd96eb41cc8f680075ec71137b P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.374048 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 23) took 3289 ms. Trace:
I20250112 02:52:37.374593 775 raft_consensus.cc:2463] T b9537fcd96eb41cc8f680075ec71137b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:37.374325 2036 rpcz_store.cc:276] 0112 02:52:34.085063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.085110 (+ 47us) service_pool.cc:225] Handling call
0112 02:52:37.348159 (+3263049us) ts_tablet_manager.cc:1145] Deleting tablet 8c821ea3ca4946c290c17135366e0cac
0112 02:52:37.348711 (+ 552us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.362692 (+ 13981us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.369749 (+ 7057us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.374030 (+ 4281us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3064778}
I20250112 02:52:37.374953 1049 raft_consensus.cc:2463] T b9537fcd96eb41cc8f680075ec71137b P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:37.375252 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 8c821ea3ca4946c290c17135366e0cac (table default.Table_0f1762619e9543018fd2df57adb6367e [id=5f5a61b2a00d4d29bd79a696474a787b]) successfully deleted
I20250112 02:52:37.375296 2036 tablet_replica.cc:331] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.375846 2036 raft_consensus.cc:2238] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.376298 854 leader_election.cc:304] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:37.376616 1173 ts_tablet_manager.cc:1428] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250112 02:52:37.376720 2222 raft_consensus.cc:491] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.376855 1173 tablet_bootstrap.cc:492] T e5c10fad790046a88fbc8ebe0303c481 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:37.376860 2222 raft_consensus.cc:513] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.377813 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e378a6a9b4a455c9f5e1d1ec0fd6ca0" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:37.378316 854 leader_election.cc:343] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.378326 2399 raft_consensus.cc:2798] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:37.378482 2399 raft_consensus.cc:695] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:37.378582 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e378a6a9b4a455c9f5e1d1ec0fd6ca0" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:37.378711 2399 consensus_queue.cc:237] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:37.378988 855 leader_election.cc:343] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.379077 855 leader_election.cc:304] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:37.379371 2117 raft_consensus.cc:2743] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:37.379549 2222 leader_election.cc:290] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:37.380340 1181 tablet_bootstrap.cc:492] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:37.380447 1181 ts_tablet_manager.cc:1397] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.061s user 0.006s sys 0.002s
W20250112 02:52:37.381289 991 consensus_peers.cc:487] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea33e5c1a1fb46599c5836a36b130051. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:37.381443 1181 raft_consensus.cc:357] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.381624 1181 raft_consensus.cc:383] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.381682 1181 raft_consensus.cc:738] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.381855 1181 consensus_queue.cc:260] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.382419 1181 ts_tablet_manager.cc:1428] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:37.382640 1181 tablet_bootstrap.cc:492] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:37.384068 1181 tablet_bootstrap.cc:654] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.387549 1177 ts_tablet_manager.cc:1428] T b9537fcd96eb41cc8f680075ec71137b P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.046s user 0.001s sys 0.001s
I20250112 02:52:37.387820 1177 tablet_bootstrap.cc:492] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:37.388993 1173 tablet_bootstrap.cc:654] T e5c10fad790046a88fbc8ebe0303c481 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.393899 2228 consensus_queue.cc:1035] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:37.394683 2024 ts_tablet_manager.cc:1918] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.395938 2024 log.cc:1198] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/aba1744138fe42ae9d097db01cac083d
I20250112 02:52:37.396070 2036 raft_consensus.cc:2267] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.397482 2179 consensus_queue.cc:1035] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:37.397658 2036 ts_tablet_manager.cc:1905] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.399016 2003 ts_tablet_manager.cc:1918] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.399145 2003 log.cc:1198] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/cc62dcbf67e84e36ae1fbe127ccd4879
I20250112 02:52:37.399574 2003 ts_tablet_manager.cc:1939] T cc62dcbf67e84e36ae1fbe127ccd4879 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:37.395835 1173 tablet_bootstrap.cc:492] T e5c10fad790046a88fbc8ebe0303c481 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:37.401845 1173 ts_tablet_manager.cc:1397] T e5c10fad790046a88fbc8ebe0303c481 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.001s
I20250112 02:52:37.402907 1173 raft_consensus.cc:357] T e5c10fad790046a88fbc8ebe0303c481 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.403527 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 55) took 3160 ms. Trace:
I20250112 02:52:37.404227 1173 raft_consensus.cc:383] T e5c10fad790046a88fbc8ebe0303c481 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.404444 1173 raft_consensus.cc:738] T e5c10fad790046a88fbc8ebe0303c481 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.404896 1173 consensus_queue.cc:260] T e5c10fad790046a88fbc8ebe0303c481 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.406093 1173 ts_tablet_manager.cc:1428] T e5c10fad790046a88fbc8ebe0303c481 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:37.406715 1173 tablet_bootstrap.cc:492] T fd7394b1878447a189ce45957f10e4df P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:37.407912 2024 ts_tablet_manager.cc:1939] T aba1744138fe42ae9d097db01cac083d P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:37.403651 2003 rpcz_store.cc:276] 0112 02:52:34.242520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.252579 (+ 10059us) service_pool.cc:225] Handling call
0112 02:52:37.322393 (+3069814us) ts_tablet_manager.cc:1145] Deleting tablet cc62dcbf67e84e36ae1fbe127ccd4879
0112 02:52:37.322404 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.337862 (+ 15458us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.398935 (+ 61073us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.403454 (+ 4519us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3066147}
I20250112 02:52:37.409752 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet cc62dcbf67e84e36ae1fbe127ccd4879 (table default.Table_fd0169db0f984d3cbc0d8624dda686bc [id=9e09b6670c6e4e57b62f31fd4755c18b]) successfully deleted
I20250112 02:52:37.409790 2003 tablet_replica.cc:331] T 1a1e3af2883341e9b957f6ed748e604d P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:37.410054 2003 raft_consensus.cc:2238] T 1a1e3af2883341e9b957f6ed748e604d P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.410245 2003 raft_consensus.cc:2267] T 1a1e3af2883341e9b957f6ed748e604d P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.410312 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 62) took 2917 ms. Trace:
I20250112 02:52:37.410761 2003 ts_tablet_manager.cc:1905] T 1a1e3af2883341e9b957f6ed748e604d P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:37.410705 991 consensus_peers.cc:487] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9163878cfeb64693a9c5493e812546af. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:37.414160 2024 rpcz_store.cc:276] 0112 02:52:34.492390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.492424 (+ 34us) service_pool.cc:225] Handling call
0112 02:52:37.343891 (+2851467us) ts_tablet_manager.cc:1145] Deleting tablet aba1744138fe42ae9d097db01cac083d
0112 02:52:37.343900 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.391097 (+ 47197us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.394609 (+ 3512us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.410295 (+ 15686us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2851328}
I20250112 02:52:37.414755 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet aba1744138fe42ae9d097db01cac083d (table default.Table_fb7e727aeed44203b2bb7e90a844e44b [id=587926735ced4c17b747f5c1fbee9f8d]) successfully deleted
I20250112 02:52:37.414819 2024 tablet_replica.cc:331] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:37.414973 2024 raft_consensus.cc:2238] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.415095 2024 raft_consensus.cc:2267] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.415544 2024 ts_tablet_manager.cc:1905] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.417344 1177 tablet_bootstrap.cc:654] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.421098 2036 ts_tablet_manager.cc:1918] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.421636 2036 log.cc:1198] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/06edfca60eed4dd3a2dee7b662b0970c
I20250112 02:52:37.422827 2036 ts_tablet_manager.cc:1939] T 06edfca60eed4dd3a2dee7b662b0970c P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.412987 1173 tablet_bootstrap.cc:654] T fd7394b1878447a189ce45957f10e4df P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.421587 2228 consensus_queue.cc:1035] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:37.423856 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 25) took 3332 ms. Trace:
I20250112 02:52:37.424197 2036 rpcz_store.cc:276] 0112 02:52:34.091219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.091278 (+ 59us) service_pool.cc:225] Handling call
0112 02:52:37.375176 (+3283898us) ts_tablet_manager.cc:1145] Deleting tablet 06edfca60eed4dd3a2dee7b662b0970c
0112 02:52:37.375185 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.407092 (+ 31907us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.420782 (+ 13690us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.423838 (+ 3056us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2,"tablet-delete.queue_time_us":3091726}
I20250112 02:52:37.425338 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 06edfca60eed4dd3a2dee7b662b0970c (table default.Table_2a2995d7964346c08fcbff96b4cc3ca5 [id=70371e156f2f401ab5ee0e4e4c7545b9]) successfully deleted
I20250112 02:52:37.425508 2036 tablet_replica.cc:331] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.426322 2036 raft_consensus.cc:2238] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.426867 2036 raft_consensus.cc:2267] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.428787 2036 ts_tablet_manager.cc:1905] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.434051 2399 raft_consensus.cc:491] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 289afc116aa74408aa5bff74e11c50d8)
I20250112 02:52:37.434172 2399 raft_consensus.cc:513] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.434764 2399 leader_election.cc:290] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:37.435254 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c639a2bcd4740e4be7b2b162004669a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:37.436214 855 leader_election.cc:343] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: 1c639a2bcd4740e4be7b2b162004669a
I20250112 02:52:37.436316 855 leader_election.cc:304] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:37.436551 713 consensus_peers.cc:487] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 352c87f1dc724e7b94bc2e64abfce8d7. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:37.436728 2399 raft_consensus.cc:2743] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:52:37.440941 2228 consensus_queue.cc:1035] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:52:37.441514 713 consensus_peers.cc:487] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1dda52f012794493a102ed1abc718f5b. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:37.446605 1177 tablet_bootstrap.cc:492] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:37.449764 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:37.450572 1181 tablet_bootstrap.cc:492] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:37.450965 2024 ts_tablet_manager.cc:1918] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.452298 1177 ts_tablet_manager.cc:1397] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.000s
I20250112 02:52:37.452951 1181 ts_tablet_manager.cc:1397] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.000s
I20250112 02:52:37.453774 1181 raft_consensus.cc:357] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.457939 2024 log.cc:1198] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/61593969d2234ce3a6089027b067d922
I20250112 02:52:37.458438 2024 ts_tablet_manager.cc:1939] T 61593969d2234ce3a6089027b067d922 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:37.460642 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 63) took 2967 ms. Trace:
I20250112 02:52:37.461650 2003 ts_tablet_manager.cc:1918] T 1a1e3af2883341e9b957f6ed748e604d P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.461794 2003 log.cc:1198] T 1a1e3af2883341e9b957f6ed748e604d P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/1a1e3af2883341e9b957f6ed748e604d
I20250112 02:52:37.458786 1177 raft_consensus.cc:357] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.463781 2024 rpcz_store.cc:276] 0112 02:52:34.492727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.492762 (+ 35us) service_pool.cc:225] Handling call
0112 02:52:37.414780 (+2922018us) ts_tablet_manager.cc:1145] Deleting tablet 61593969d2234ce3a6089027b067d922
0112 02:52:37.414789 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.430177 (+ 15388us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.450853 (+ 20676us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.460614 (+ 9761us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2921841}
I20250112 02:52:37.464267 2024 tablet_replica.cc:331] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:37.465093 2024 raft_consensus.cc:2238] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:37.467564 1181 raft_consensus.cc:383] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.467644 1181 raft_consensus.cc:738] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.467687 2024 raft_consensus.cc:2267] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.467862 1181 consensus_queue.cc:260] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.468422 1181 ts_tablet_manager.cc:1428] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20250112 02:52:37.468663 1181 tablet_bootstrap.cc:492] T 6171b8ff2bfe41d598b0cb2535799a11 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:37.468787 2024 ts_tablet_manager.cc:1905] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.470136 1181 tablet_bootstrap.cc:654] T 6171b8ff2bfe41d598b0cb2535799a11 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.452093 966 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:37.460489 2036 ts_tablet_manager.cc:1918] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.470801 2036 log.cc:1198] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/4348fe4c32704bf7b3a8d21ff3b8ac1b
I20250112 02:52:37.471303 2036 ts_tablet_manager.cc:1939] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.472077 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 26) took 3379 ms. Trace:
I20250112 02:52:37.472337 2036 rpcz_store.cc:276] 0112 02:52:34.092177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.092238 (+ 61us) service_pool.cc:225] Handling call
0112 02:52:37.425318 (+3333080us) ts_tablet_manager.cc:1145] Deleting tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b
0112 02:52:37.425328 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.452349 (+ 27021us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.460200 (+ 7851us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.471734 (+ 11534us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3141826}
I20250112 02:52:37.473282 1173 tablet_bootstrap.cc:492] T fd7394b1878447a189ce45957f10e4df P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:37.473294 2036 tablet_replica.cc:331] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.473475 1173 ts_tablet_manager.cc:1397] T fd7394b1878447a189ce45957f10e4df P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.067s user 0.007s sys 0.000s
I20250112 02:52:37.473555 2036 raft_consensus.cc:2238] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.473731 2036 raft_consensus.cc:2267] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.474431 2036 ts_tablet_manager.cc:1905] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.475540 1177 raft_consensus.cc:383] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.475425 1173 raft_consensus.cc:357] T fd7394b1878447a189ce45957f10e4df P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.475637 1177 raft_consensus.cc:738] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.475785 1173 raft_consensus.cc:383] T fd7394b1878447a189ce45957f10e4df P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.475915 1173 raft_consensus.cc:738] T fd7394b1878447a189ce45957f10e4df P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.475843 1177 consensus_queue.cc:260] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.476369 1177 ts_tablet_manager.cc:1428] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.024s user 0.001s sys 0.000s
I20250112 02:52:37.476287 1173 consensus_queue.cc:260] T fd7394b1878447a189ce45957f10e4df P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.476610 1177 tablet_bootstrap.cc:492] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:37.476893 1173 ts_tablet_manager.cc:1428] T fd7394b1878447a189ce45957f10e4df P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 02:52:37.477216 1173 tablet_bootstrap.cc:492] T e933b58dab0b45e8bf1c49eebb1c53e3 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:37.478578 2222 raft_consensus.cc:491] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.478690 2222 raft_consensus.cc:513] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.479343 2222 leader_election.cc:290] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:37.480096 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4ef0220dde43569452379639759db5" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:37.480223 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4ef0220dde43569452379639759db5" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:37.480549 855 leader_election.cc:343] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:37.480651 854 leader_election.cc:343] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.480741 854 leader_election.cc:304] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:37.481110 2222 raft_consensus.cc:2743] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:37.481602 1177 tablet_bootstrap.cc:654] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.484488 1173 tablet_bootstrap.cc:654] T e933b58dab0b45e8bf1c49eebb1c53e3 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.485181 2036 ts_tablet_manager.cc:1918] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.485597 2036 log.cc:1198] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/b9c006281dc949119951c74e23167d7f
I20250112 02:52:37.486941 2036 ts_tablet_manager.cc:1939] T b9c006281dc949119951c74e23167d7f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.490360 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 27) took 3396 ms. Trace:
I20250112 02:52:37.464264 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 61593969d2234ce3a6089027b067d922 (table default.Table_0188036e1bd0455ea82fd355934295fc [id=b2f29883bc3844cf8b008345b849345c]) successfully deleted
I20250112 02:52:37.473052 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b (table default.Table_1d107b48a09e41d6bb3401b01a21ee78 [id=68bab352836142c39c109e8e02553802]) successfully deleted
I20250112 02:52:37.492415 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:37.492796 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:37.492942 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:37.493198 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
I20250112 02:52:37.493396 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet f3771166b77e4f90aaa116ab1a9dc299 (table default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6]). Expected version 1 got 0
I20250112 02:52:37.493620 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:37.490757 2036 rpcz_store.cc:276] 0112 02:52:34.093305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.093358 (+ 53us) service_pool.cc:225] Handling call
0112 02:52:37.473229 (+3379871us) ts_tablet_manager.cc:1145] Deleting tablet b9c006281dc949119951c74e23167d7f
0112 02:52:37.473237 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.478693 (+ 5456us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.484914 (+ 6221us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.490244 (+ 5330us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3189758}
I20250112 02:52:37.494299 2222 raft_consensus.cc:491] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.494421 2222 raft_consensus.cc:513] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.494515 2036 tablet_replica.cc:331] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.494618 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet b9c006281dc949119951c74e23167d7f (table default.Table_1a6ba73a53b14addbacc6a2ff0711256 [id=fe95e997b21346a5b36675a9c4d25a06]) successfully deleted
I20250112 02:52:37.494812 2036 raft_consensus.cc:2238] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.495000 2222 leader_election.cc:290] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:37.495020 2036 raft_consensus.cc:2267] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.494868 447 catalog_manager.cc:5526] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:37.495398 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet fccff64af8ca4400afc116db98918119 (table default.Table_15bcb2c0f2a74458a1bed231fd3437fc [id=ca9c8281bf254f248076cb600e7c4f87]). Expected version 2 got 0
I20250112 02:52:37.495738 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet f924459d1150472aafb325a11825db23 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]). Expected version 3 got 0
I20250112 02:52:37.496174 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 09d06e2e8c13464db8cf801a292bdc99 (table default.Table_0bdc5f49d4ff4d6fb1290bc2dc0f70d9 [id=e3bb489c53454222a79702b798c0c2f1]). Expected version 2 got 0
I20250112 02:52:37.496204 2036 ts_tablet_manager.cc:1905] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.496814 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a0d939fdb94590a86389f1f4693e9a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:37.497798 854 leader_election.cc:343] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.498333 2228 consensus_queue.cc:1035] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:37.462203 2003 ts_tablet_manager.cc:1939] T 1a1e3af2883341e9b957f6ed748e604d P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:37.496189 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a0d939fdb94590a86389f1f4693e9a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:37.508349 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 54) took 3266 ms. Trace:
W20250112 02:52:37.508435 855 leader_election.cc:343] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.508529 855 leader_election.cc:304] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:37.509011 2222 raft_consensus.cc:2743] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:37.511281 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:37.511533 2003 rpcz_store.cc:276] 0112 02:52:34.241941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.257613 (+ 15672us) service_pool.cc:225] Handling call
0112 02:52:37.409713 (+3152100us) ts_tablet_manager.cc:1145] Deleting tablet 1a1e3af2883341e9b957f6ed748e604d
0112 02:52:37.409723 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.425807 (+ 16084us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.461514 (+ 35707us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.508305 (+ 46791us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3151507}
I20250112 02:52:37.511992 2003 tablet_replica.cc:331] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:37.512310 2003 raft_consensus.cc:2238] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:37.512684 2003 raft_consensus.cc:2267] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.513285 2003 ts_tablet_manager.cc:1905] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.514456 1173 tablet_bootstrap.cc:492] T e933b58dab0b45e8bf1c49eebb1c53e3 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:37.514585 1173 ts_tablet_manager.cc:1397] T e933b58dab0b45e8bf1c49eebb1c53e3 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.003s
I20250112 02:52:37.515153 2036 ts_tablet_manager.cc:1918] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.515172 1173 raft_consensus.cc:357] T e933b58dab0b45e8bf1c49eebb1c53e3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.515293 2036 log.cc:1198] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/2730634a5cfd4c2fb5c4e30100c556d4
I20250112 02:52:37.515331 1173 raft_consensus.cc:383] T e933b58dab0b45e8bf1c49eebb1c53e3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.515381 1173 raft_consensus.cc:738] T e933b58dab0b45e8bf1c49eebb1c53e3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.515887 2036 ts_tablet_manager.cc:1939] T 2730634a5cfd4c2fb5c4e30100c556d4 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.517036 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 29) took 3411 ms. Trace:
I20250112 02:52:37.517448 2036 rpcz_store.cc:276] 0112 02:52:34.105094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.105160 (+ 66us) service_pool.cc:225] Handling call
0112 02:52:37.494448 (+3389288us) ts_tablet_manager.cc:1145] Deleting tablet 2730634a5cfd4c2fb5c4e30100c556d4
0112 02:52:37.494479 (+ 31us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.510851 (+ 16372us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.515063 (+ 4212us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.517017 (+ 1954us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3210908}
I20250112 02:52:37.517956 2036 tablet_replica.cc:331] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.518205 2036 raft_consensus.cc:2238] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.518362 2036 raft_consensus.cc:2267] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.518972 2024 ts_tablet_manager.cc:1918] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.518990 2036 ts_tablet_manager.cc:1905] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.519057 2024 log.cc:1198] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/65f22a7b1b804661af3b7e785a833739
I20250112 02:52:37.519347 2024 ts_tablet_manager.cc:1939] T 65f22a7b1b804661af3b7e785a833739 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:37.522540 1173 consensus_queue.cc:260] T e933b58dab0b45e8bf1c49eebb1c53e3 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:37.523690 714 consensus_peers.cc:487] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:37.511947 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 1a1e3af2883341e9b957f6ed748e604d (table default.Table_b3ed6c7a79ed43568b8743f8aea9f67e [id=902e40c3f1e9401e934a4a2652f3dc22]) successfully deleted
I20250112 02:52:37.517982 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 2730634a5cfd4c2fb5c4e30100c556d4 (table default.Table_a98f3e8602ec4ba4a771b14df9c07131 [id=274b4ed84aa7490dbb78270b48150b73]) successfully deleted
I20250112 02:52:37.528306 1173 ts_tablet_manager.cc:1428] T e933b58dab0b45e8bf1c49eebb1c53e3 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250112 02:52:37.528497 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:37.528606 1173 tablet_bootstrap.cc:492] T 4d25815a5df94e099e7481c48c3da793 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:37.529089 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet d8eb0efc5d7f4c72bdfc7531a444d42a (table default.Table_d561e1740cc545dc8aa4babb76b45c37 [id=de27aeff6b3a49b3956498ea5015ff33]). Expected version 1 got 0
I20250112 02:52:37.529095 2036 ts_tablet_manager.cc:1918] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.529192 2036 log.cc:1198] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/b61f16f3d601436bb4b32344de28c1a7
I20250112 02:52:37.529645 2036 ts_tablet_manager.cc:1939] T b61f16f3d601436bb4b32344de28c1a7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.529840 1173 tablet_bootstrap.cc:654] T 4d25815a5df94e099e7481c48c3da793 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.530591 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 64) took 3037 ms. Trace:
I20250112 02:52:37.530709 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 28) took 3426 ms. Trace:
I20250112 02:52:37.530666 2024 rpcz_store.cc:276] 0112 02:52:34.493119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.493201 (+ 82us) service_pool.cc:225] Handling call
0112 02:52:37.464229 (+2971028us) ts_tablet_manager.cc:1145] Deleting tablet 65f22a7b1b804661af3b7e785a833739
0112 02:52:37.464237 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.501032 (+ 36795us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.518879 (+ 17847us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.530575 (+ 11696us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2970875}
I20250112 02:52:37.530840 2036 rpcz_store.cc:276] 0112 02:52:34.104100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.105333 (+ 1233us) service_pool.cc:225] Handling call
0112 02:52:37.517863 (+3412530us) ts_tablet_manager.cc:1145] Deleting tablet b61f16f3d601436bb4b32344de28c1a7
0112 02:52:37.517896 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.524572 (+ 6676us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.529000 (+ 4428us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.530691 (+ 1691us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3234286}
I20250112 02:52:37.531275 2036 tablet_replica.cc:331] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.531566 2036 raft_consensus.cc:2238] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.531795 1181 tablet_bootstrap.cc:492] T 6171b8ff2bfe41d598b0cb2535799a11 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:37.531770 2036 raft_consensus.cc:2267] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.531983 1181 ts_tablet_manager.cc:1397] T 6171b8ff2bfe41d598b0cb2535799a11 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.001s
I20250112 02:52:37.532588 2036 ts_tablet_manager.cc:1905] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.532725 1181 raft_consensus.cc:357] T 6171b8ff2bfe41d598b0cb2535799a11 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.532924 1181 raft_consensus.cc:383] T 6171b8ff2bfe41d598b0cb2535799a11 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.533015 1181 raft_consensus.cc:738] T 6171b8ff2bfe41d598b0cb2535799a11 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.533257 1181 consensus_queue.cc:260] T 6171b8ff2bfe41d598b0cb2535799a11 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.536667 1181 ts_tablet_manager.cc:1428] T 6171b8ff2bfe41d598b0cb2535799a11 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250112 02:52:37.536959 1181 tablet_bootstrap.cc:492] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:37.537654 2024 tablet_replica.cc:331] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:37.537837 2024 raft_consensus.cc:2238] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.537994 2024 raft_consensus.cc:2267] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.538467 2024 ts_tablet_manager.cc:1905] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.538905 1181 tablet_bootstrap.cc:654] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.541522 2036 ts_tablet_manager.cc:1918] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.541700 2036 log.cc:1198] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/7a18132549374218893f35a4e4226b72
W20250112 02:52:37.542274 991 consensus_peers.cc:487] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b599e983ec04f53a440d1ab828d12ab. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:37.542312 2036 ts_tablet_manager.cc:1939] T 7a18132549374218893f35a4e4226b72 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.542809 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 30) took 3418 ms. Trace:
I20250112 02:52:37.542934 2036 rpcz_store.cc:276] 0112 02:52:34.124564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.124617 (+ 53us) service_pool.cc:225] Handling call
0112 02:52:37.531220 (+3406603us) ts_tablet_manager.cc:1145] Deleting tablet 7a18132549374218893f35a4e4226b72
0112 02:52:37.531229 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.536807 (+ 5578us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.541399 (+ 4592us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.542778 (+ 1379us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":49920,"tablet-delete.queue_time_us":3246950}
I20250112 02:52:37.543511 2036 tablet_replica.cc:331] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.543910 2036 raft_consensus.cc:2238] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.544131 2036 raft_consensus.cc:2267] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.544556 1177 tablet_bootstrap.cc:492] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:37.544744 1177 ts_tablet_manager.cc:1397] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.002s
I20250112 02:52:37.544844 2036 ts_tablet_manager.cc:1905] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.545540 1177 raft_consensus.cc:357] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.545797 1177 raft_consensus.cc:383] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.545900 1177 raft_consensus.cc:738] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.546203 1177 consensus_queue.cc:260] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.546836 1177 ts_tablet_manager.cc:1428] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:37.550388 2317 consensus_queue.cc:1035] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:37.556433 2317 raft_consensus.cc:491] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.557062 1173 tablet_bootstrap.cc:492] T 4d25815a5df94e099e7481c48c3da793 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:37.557168 1173 ts_tablet_manager.cc:1397] T 4d25815a5df94e099e7481c48c3da793 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20250112 02:52:37.531026 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 65f22a7b1b804661af3b7e785a833739 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]) successfully deleted
I20250112 02:52:37.531337 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet b61f16f3d601436bb4b32344de28c1a7 (table default.Table_dafd4ddc8ab34adfa999f9340216f765 [id=da4d2d7725904d5bb4a6bde668df00ef]) successfully deleted
I20250112 02:52:37.556599 2317 raft_consensus.cc:513] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.557660 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 7a18132549374218893f35a4e4226b72 (table default.Table_3fcca85a33054d7ab388f29bba85fe35 [id=18415100e94043e99f9884b853158ae1]) successfully deleted
I20250112 02:52:37.557816 1173 raft_consensus.cc:357] T 4d25815a5df94e099e7481c48c3da793 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.557963 1173 raft_consensus.cc:383] T 4d25815a5df94e099e7481c48c3da793 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.558035 1173 raft_consensus.cc:738] T 4d25815a5df94e099e7481c48c3da793 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.558228 1173 consensus_queue.cc:260] T 4d25815a5df94e099e7481c48c3da793 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.558452 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a7281c0c09442ebd0cc13b8496c86e" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:37.558531 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
I20250112 02:52:37.558624 1049 raft_consensus.cc:2463] T c8a7281c0c09442ebd0cc13b8496c86e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:37.558709 1173 ts_tablet_manager.cc:1428] T 4d25815a5df94e099e7481c48c3da793 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 02:52:37.558715 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:37.558923 1173 tablet_bootstrap.cc:492] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:37.559152 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:37.559409 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:37.559659 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 6117dc8b32a64535a2f4167a03bed0d2 (table default.Table_365739770a11434dbf260ea912e7ea81 [id=868adb627654450c936ef198e49d853b]). Expected version 1 got 0
I20250112 02:52:37.559927 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 2c03b5e612d64314b210c11686fdfa2a (table default.Table_4934f6022f2441eda88751236a5798be [id=5c4769bfcefd4fe0a0f3c2e315d50c32]). Expected version 6 got 0
I20250112 02:52:37.560184 2222 raft_consensus.cc:491] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.560240 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:37.560281 2222 raft_consensus.cc:513] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.560381 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3f5b418c17084420b867498fda79fb0a (table default.Table_96d1a09a902340ff8e05efb75faf3854 [id=ad0be37e6b6f4b9597dcb4cbd50d5b12]). Expected version 3 got 2
I20250112 02:52:37.560525 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:37.560684 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:37.560739 2222 leader_election.cc:290] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:37.560971 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:37.561220 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:37.561601 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70715db0394d46d9b6b289160b00be7d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:37.561815 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 5cc414d6420c41839621ab858067548b (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]). Expected version 1 got 0
W20250112 02:52:37.562034 854 leader_election.cc:343] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.562256 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:37.562773 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:37.565076 1173 tablet_bootstrap.cc:654] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.565944 2317 leader_election.cc:290] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:37.565924 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70715db0394d46d9b6b289160b00be7d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:37.566370 855 leader_election.cc:343] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.566470 855 leader_election.cc:304] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:37.566704 2222 raft_consensus.cc:2743] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:37.567500 713 leader_election.cc:343] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: c8a7281c0c09442ebd0cc13b8496c86e
I20250112 02:52:37.568065 2024 ts_tablet_manager.cc:1918] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.568238 2024 log.cc:1198] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/b7bd96fb381f4e73a0081be3fc799c36
I20250112 02:52:37.568622 2024 ts_tablet_manager.cc:1939] T b7bd96fb381f4e73a0081be3fc799c36 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:37.568675 714 leader_election.cc:304] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:37.568928 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 65) took 3075 ms. Trace:
I20250112 02:52:37.569031 2317 raft_consensus.cc:2798] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:37.569115 2317 raft_consensus.cc:491] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:37.569209 2317 raft_consensus.cc:3054] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.572039 2024 rpcz_store.cc:276] 0112 02:52:34.493504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.509141 (+ 15637us) service_pool.cc:225] Handling call
0112 02:52:37.537617 (+3028476us) ts_tablet_manager.cc:1145] Deleting tablet b7bd96fb381f4e73a0081be3fc799c36
0112 02:52:37.537626 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.550543 (+ 12917us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.567992 (+ 17449us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.568914 (+ 922us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3028257}
I20250112 02:52:37.572757 2222 raft_consensus.cc:491] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.572870 2222 raft_consensus.cc:513] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.573364 2222 leader_election.cc:290] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:37.572520 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet b7bd96fb381f4e73a0081be3fc799c36 (table default.Table_893bf955f62b495c9dcc59390b472f96 [id=e992deac51c64dd8a022428dd3c53714]) successfully deleted
I20250112 02:52:37.574011 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead501d76bb1412e9839fb36b9911e6f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:37.574426 855 leader_election.cc:343] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.574457 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 48634d6d78784fae98512b242e5476df (table default.Table_7cbe570a89784c18b4a14a8242062071 [id=fee2f9e65e314ce9a1fc1fb6dfcde317]). Expected version 1 got 0
I20250112 02:52:37.574507 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead501d76bb1412e9839fb36b9911e6f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:37.574904 854 leader_election.cc:343] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.574988 854 leader_election.cc:304] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:37.576335 2399 raft_consensus.cc:2743] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:37.574685 447 catalog_manager.cc:5526] T 89b70dce4d1d4e29997a18297ac0f82a P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:37.577569 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:37.577745 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet f31b777e7e0f43a28839d8c48c6a6590 (table default.Table_5fb592e41cc4406aa27546e7b41fdb88 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 1 got 0
I20250112 02:52:37.577962 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:37.578119 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:37.578378 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:37.578552 2317 raft_consensus.cc:513] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.579125 2024 tablet_replica.cc:331] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:37.579281 2024 raft_consensus.cc:2238] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.579397 2024 raft_consensus.cc:2267] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.579393 2317 leader_election.cc:290] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:37.579829 2024 ts_tablet_manager.cc:1905] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:37.580152 713 leader_election.cc:343] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: c8a7281c0c09442ebd0cc13b8496c86e
I20250112 02:52:37.580775 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a7281c0c09442ebd0cc13b8496c86e" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:37.580935 1049 raft_consensus.cc:3054] T c8a7281c0c09442ebd0cc13b8496c86e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.582101 1181 tablet_bootstrap.cc:492] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:37.582207 1181 ts_tablet_manager.cc:1397] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.045s user 0.001s sys 0.002s
I20250112 02:52:37.582934 1181 raft_consensus.cc:357] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.583087 1181 raft_consensus.cc:383] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.583140 1181 raft_consensus.cc:738] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.583284 1049 raft_consensus.cc:2463] T c8a7281c0c09442ebd0cc13b8496c86e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:37.583374 1181 consensus_queue.cc:260] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.583832 714 leader_election.cc:304] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:37.583879 1181 ts_tablet_manager.cc:1428] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:37.584081 1181 tablet_bootstrap.cc:492] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:37.584537 2047 raft_consensus.cc:2798] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:37.584674 2047 raft_consensus.cc:695] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:37.584893 2047 consensus_queue.cc:237] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.585536 2024 ts_tablet_manager.cc:1918] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.585634 2024 log.cc:1198] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/afa2f8aabe5e42ccbc15b193fdc03563
I20250112 02:52:37.585718 1181 tablet_bootstrap.cc:654] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.585883 2024 ts_tablet_manager.cc:1939] T afa2f8aabe5e42ccbc15b193fdc03563 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:37.586199 1177 tablet_bootstrap.cc:492] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:37.588883 1177 tablet_bootstrap.cc:654] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.589354 2036 ts_tablet_manager.cc:1918] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.590749 2036 log.cc:1198] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/d45fce86e02f46d4a57af718fa2a62c8
I20250112 02:52:37.587034 1173 tablet_bootstrap.cc:492] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:37.590982 1173 ts_tablet_manager.cc:1397] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20250112 02:52:37.591629 1173 raft_consensus.cc:357] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.591791 1173 raft_consensus.cc:383] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.591842 1173 raft_consensus.cc:738] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.592031 1173 consensus_queue.cc:260] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.601610 2036 ts_tablet_manager.cc:1939] T d45fce86e02f46d4a57af718fa2a62c8 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.602186 1173 ts_tablet_manager.cc:1428] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20250112 02:52:37.602442 1173 tablet_bootstrap.cc:492] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:37.603351 1181 tablet_bootstrap.cc:492] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:37.603487 1181 ts_tablet_manager.cc:1397] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.001s
I20250112 02:52:37.604049 1181 raft_consensus.cc:357] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.604203 1181 raft_consensus.cc:383] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.604254 1181 raft_consensus.cc:738] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.604408 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 31) took 3478 ms. Trace:
I20250112 02:52:37.604964 2036 rpcz_store.cc:276] 0112 02:52:34.125583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.258485 (+132902us) service_pool.cc:225] Handling call
0112 02:52:37.543420 (+3284935us) ts_tablet_manager.cc:1145] Deleting tablet d45fce86e02f46d4a57af718fa2a62c8
0112 02:52:37.543431 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.562256 (+ 18825us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.588807 (+ 26551us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.604287 (+ 15480us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3229798}
I20250112 02:52:37.606256 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet d45fce86e02f46d4a57af718fa2a62c8 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]) successfully deleted
I20250112 02:52:37.604437 1181 consensus_queue.cc:260] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.606894 1181 ts_tablet_manager.cc:1428] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 02:52:37.607002 2036 tablet_replica.cc:331] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.607111 1181 tablet_bootstrap.cc:492] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:37.608120 2036 raft_consensus.cc:2238] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.609138 1173 tablet_bootstrap.cc:654] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.609088 2036 raft_consensus.cc:2267] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.609684 1181 tablet_bootstrap.cc:654] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.610457 1102 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:37.612874 2003 ts_tablet_manager.cc:1918] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.613058 2003 log.cc:1198] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/04f135b226774285a077e5806131d91b
I20250112 02:52:37.613756 2003 ts_tablet_manager.cc:1939] T 04f135b226774285a077e5806131d91b P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:37.613925 2036 ts_tablet_manager.cc:1905] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.618000 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 66) took 3108 ms. Trace:
I20250112 02:52:37.618085 2024 rpcz_store.cc:276] 0112 02:52:34.509516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.513042 (+ 3526us) service_pool.cc:225] Handling call
0112 02:52:37.572318 (+3059276us) ts_tablet_manager.cc:1145] Deleting tablet afa2f8aabe5e42ccbc15b193fdc03563
0112 02:52:37.578810 (+ 6492us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.583066 (+ 4256us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.585476 (+ 2410us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.617984 (+ 32508us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3032940}
I20250112 02:52:37.618438 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet afa2f8aabe5e42ccbc15b193fdc03563 (table default.Table_e21cbee8871f4ab583b22a3eaa4b5e6a [id=1b3ac0e8c0704c618077010dda4f7413]) successfully deleted
I20250112 02:52:37.620185 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 80) took 3360 ms. Trace:
I20250112 02:52:37.620756 2024 tablet_replica.cc:331] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:37.620929 2024 raft_consensus.cc:2238] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.621037 2024 raft_consensus.cc:2267] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.621481 2024 ts_tablet_manager.cc:1905] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.621665 2003 rpcz_store.cc:276] 0112 02:52:34.259572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.259611 (+ 39us) service_pool.cc:225] Handling call
0112 02:52:37.511933 (+3252322us) ts_tablet_manager.cc:1145] Deleting tablet 04f135b226774285a077e5806131d91b
0112 02:52:37.511942 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.516583 (+ 4641us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.612768 (+ 96185us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.620139 (+ 7371us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3252116}
I20250112 02:52:37.622180 2003 tablet_replica.cc:331] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:37.622560 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 04f135b226774285a077e5806131d91b (table default.Table_d1e5cef5fa3e4d90bada176f80bd4180 [id=f93b5bb22ec1424ea047383ed9713618]) successfully deleted
I20250112 02:52:37.624891 2036 ts_tablet_manager.cc:1918] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.625032 2036 log.cc:1198] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/161fe26b9c0e4543b7aeb0021506d72a
I20250112 02:52:37.625629 1173 tablet_bootstrap.cc:492] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:37.625737 1173 ts_tablet_manager.cc:1397] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.002s
I20250112 02:52:37.625660 2003 raft_consensus.cc:2238] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:37.625747 2036 ts_tablet_manager.cc:1939] T 161fe26b9c0e4543b7aeb0021506d72a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.614405 1177 tablet_bootstrap.cc:492] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:37.626308 1173 raft_consensus.cc:357] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.626470 1173 raft_consensus.cc:383] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.626518 1173 raft_consensus.cc:738] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.626623 1177 ts_tablet_manager.cc:1397] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.000s
I20250112 02:52:37.626708 1173 consensus_queue.cc:260] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:37.627187 713 consensus_peers.cc:487] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2730634a5cfd4c2fb5c4e30100c556d4. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:37.628476 1177 raft_consensus.cc:357] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.628736 1177 raft_consensus.cc:383] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.628870 1177 raft_consensus.cc:738] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.629119 1177 consensus_queue.cc:260] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.629581 1173 ts_tablet_manager.cc:1428] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 02:52:37.629904 1173 tablet_bootstrap.cc:492] T 6cb76746b3a846c3aba531c04bf63966 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:37.631371 1173 tablet_bootstrap.cc:654] T 6cb76746b3a846c3aba531c04bf63966 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.631492 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 72) took 3263 ms. Trace:
I20250112 02:52:37.631633 2036 rpcz_store.cc:276] 0112 02:52:34.368223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.368387 (+ 164us) service_pool.cc:225] Handling call
0112 02:52:37.606749 (+3238362us) ts_tablet_manager.cc:1145] Deleting tablet 161fe26b9c0e4543b7aeb0021506d72a
0112 02:52:37.606759 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.619517 (+ 12758us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.624779 (+ 5262us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.631446 (+ 6667us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3212192}
I20250112 02:52:37.632179 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 161fe26b9c0e4543b7aeb0021506d72a (table default.Table_981a70d4d3184fd8a99e6b81ae1caab7 [id=8016eb490f574ad8a5d3ee598fecaeb5]) successfully deleted
I20250112 02:52:37.632248 2036 tablet_replica.cc:331] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.632531 2036 raft_consensus.cc:2238] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.632743 2036 raft_consensus.cc:2267] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.633520 2036 ts_tablet_manager.cc:1905] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.635058 1177 ts_tablet_manager.cc:1428] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250112 02:52:37.635558 1177 tablet_bootstrap.cc:492] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:37.644804 2036 ts_tablet_manager.cc:1918] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.644933 2036 log.cc:1198] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/416fb6820cd54e118950c68d381f51b2
I20250112 02:52:37.645406 2036 ts_tablet_manager.cc:1939] T 416fb6820cd54e118950c68d381f51b2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.645826 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 73) took 3277 ms. Trace:
I20250112 02:52:37.645921 2036 rpcz_store.cc:276] 0112 02:52:34.368348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.368513 (+ 165us) service_pool.cc:225] Handling call
0112 02:52:37.632177 (+3263664us) ts_tablet_manager.cc:1145] Deleting tablet 416fb6820cd54e118950c68d381f51b2
0112 02:52:37.632188 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.639514 (+ 7326us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.644701 (+ 5187us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.645787 (+ 1086us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3237802}
I20250112 02:52:37.646544 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 416fb6820cd54e118950c68d381f51b2 (table default.Table_f4d612cd6a524bc493147493c5a4a284 [id=4ac16bc62bbe4ed7a977d22c92ad580d]) successfully deleted
I20250112 02:52:37.646579 2036 tablet_replica.cc:331] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.646831 2036 raft_consensus.cc:2238] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:37.647342 2036 raft_consensus.cc:2267] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.648357 2036 ts_tablet_manager.cc:1905] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.630961 2003 pending_rounds.cc:62] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:37.635704 2445 consensus_queue.cc:1035] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:37.648514 2003 pending_rounds.cc:69] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319860902563840 op_type: NO_OP noop_request { }
I20250112 02:52:37.648666 2003 raft_consensus.cc:2883] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:37.648783 2003 raft_consensus.cc:2267] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.640789 2317 raft_consensus.cc:491] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.651084 2317 raft_consensus.cc:513] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.654223 2317 leader_election.cc:290] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:37.655771 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7394b1878447a189ce45957f10e4df" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:37.656052 1049 raft_consensus.cc:2463] T fd7394b1878447a189ce45957f10e4df P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:37.656560 714 leader_election.cc:304] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:37.656787 2445 raft_consensus.cc:2798] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:37.656862 2445 raft_consensus.cc:491] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:37.656919 2445 raft_consensus.cc:3054] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.659273 2445 raft_consensus.cc:513] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.660548 2445 leader_election.cc:290] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:37.662849 1177 tablet_bootstrap.cc:654] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.665482 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7394b1878447a189ce45957f10e4df" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:37.665777 1049 raft_consensus.cc:3054] T fd7394b1878447a189ce45957f10e4df P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.667110 1051 raft_consensus.cc:3054] T e933b58dab0b45e8bf1c49eebb1c53e3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.671069 1049 raft_consensus.cc:2463] T fd7394b1878447a189ce45957f10e4df P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:37.654060 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7394b1878447a189ce45957f10e4df" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:37.660634 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7394b1878447a189ce45957f10e4df" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:37.672632 910 raft_consensus.cc:2463] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:37.664777 1181 tablet_bootstrap.cc:492] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:37.667961 2399 raft_consensus.cc:491] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.672932 1181 ts_tablet_manager.cc:1397] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.005s
I20250112 02:52:37.672907 2399 raft_consensus.cc:513] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.673485 2399 leader_election.cc:290] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:37.673779 1181 raft_consensus.cc:357] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.673995 1181 raft_consensus.cc:383] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.674069 1181 raft_consensus.cc:738] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.674260 1181 consensus_queue.cc:260] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.674803 1181 ts_tablet_manager.cc:1428] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:37.675115 1181 tablet_bootstrap.cc:492] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:37.675532 2024 ts_tablet_manager.cc:1918] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.676721 1173 tablet_bootstrap.cc:492] T 6cb76746b3a846c3aba531c04bf63966 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:37.676913 1173 ts_tablet_manager.cc:1397] T 6cb76746b3a846c3aba531c04bf63966 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20250112 02:52:37.677227 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42444e99aba84a7b873bc8cdf954a6dc" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:37.677583 1173 raft_consensus.cc:357] T 6cb76746b3a846c3aba531c04bf63966 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:37.677706 854 leader_election.cc:343] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.677760 1173 raft_consensus.cc:383] T 6cb76746b3a846c3aba531c04bf63966 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 02:52:37.671248 713 consensus_peers.cc:487] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b184806b4884cd3b0a002905a1e57af. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:37.677807 1173 raft_consensus.cc:738] T 6cb76746b3a846c3aba531c04bf63966 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.678140 1181 tablet_bootstrap.cc:654] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.678011 1173 consensus_queue.cc:260] T 6cb76746b3a846c3aba531c04bf63966 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.671638 714 leader_election.cc:304] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:37.675724 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42444e99aba84a7b873bc8cdf954a6dc" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:37.680266 2208 raft_consensus.cc:491] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.680377 2208 raft_consensus.cc:513] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.680893 2208 leader_election.cc:290] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:37.681558 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c5a861a90840d0a33b780ec75bffd2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:37.681725 775 raft_consensus.cc:2463] T b3c5a861a90840d0a33b780ec75bffd2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:37.681929 1173 ts_tablet_manager.cc:1428] T 6cb76746b3a846c3aba531c04bf63966 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250112 02:52:37.682085 991 leader_election.cc:304] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:37.682159 1173 tablet_bootstrap.cc:492] T d45fce86e02f46d4a57af718fa2a62c8 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:37.682518 2208 raft_consensus.cc:2798] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:37.682585 2208 raft_consensus.cc:491] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:37.682639 2208 raft_consensus.cc:3054] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.683531 2445 raft_consensus.cc:2798] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:37.683648 2445 raft_consensus.cc:695] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:37.683661 1173 tablet_bootstrap.cc:654] T d45fce86e02f46d4a57af718fa2a62c8 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.683827 2445 consensus_queue.cc:237] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.684653 2317 consensus_queue.cc:1035] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:37.685034 2208 raft_consensus.cc:513] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.686619 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c5a861a90840d0a33b780ec75bffd2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:37.686810 775 raft_consensus.cc:3054] T b3c5a861a90840d0a33b780ec75bffd2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.689615 2024 log.cc:1198] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/1726d61cc36a4deb83c5d604e8daf4f0
I20250112 02:52:37.689883 2024 ts_tablet_manager.cc:1939] T 1726d61cc36a4deb83c5d604e8daf4f0 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:37.690414 775 raft_consensus.cc:2463] T b3c5a861a90840d0a33b780ec75bffd2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:37.690907 991 leader_election.cc:304] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:37.691129 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 67) took 3177 ms. Trace:
W20250112 02:52:37.678680 855 leader_election.cc:343] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.681277 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c5a861a90840d0a33b780ec75bffd2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:37.695667 910 raft_consensus.cc:2463] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:37.691365 2399 raft_consensus.cc:491] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.696349 2399 raft_consensus.cc:513] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.696846 2399 leader_election.cc:290] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:37.692252 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c5a861a90840d0a33b780ec75bffd2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:37.697021 919 raft_consensus.cc:3054] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.699074 919 raft_consensus.cc:2463] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:37.691139 2326 raft_consensus.cc:2798] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:37.699410 2326 raft_consensus.cc:695] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:37.699648 2326 consensus_queue.cc:237] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.692473 2208 leader_election.cc:290] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:37.696301 2228 consensus_queue.cc:1035] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:37.699339 2024 rpcz_store.cc:276] 0112 02:52:34.513388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.514345 (+ 957us) service_pool.cc:225] Handling call
0112 02:52:37.618306 (+3103961us) ts_tablet_manager.cc:1145] Deleting tablet 1726d61cc36a4deb83c5d604e8daf4f0
0112 02:52:37.618318 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.660126 (+ 41808us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.662966 (+ 2840us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.691116 (+ 28150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3078774}
I20250112 02:52:37.703949 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 1726d61cc36a4deb83c5d604e8daf4f0 (table default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6]) successfully deleted
I20250112 02:52:37.704600 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 88) took 3169 ms. Trace:
I20250112 02:52:37.705173 2024 rpcz_store.cc:276] 0112 02:52:34.534806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.539549 (+ 4743us) service_pool.cc:225] Handling call
0112 02:52:37.704459 (+3164910us) ts_tablet_manager.cc:1145] Deleting tablet b6d84027ad6d40a5be5afcdd854431a5
0112 02:52:37.704476 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.704583 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3164057}
W20250112 02:52:37.705992 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b6d84027ad6d40a5be5afcdd854431a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6d84027ad6d40a5be5afcdd854431a5 already in progress: creating tablet
I20250112 02:52:37.706451 2024 tablet_replica.cc:331] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:37.706941 2024 raft_consensus.cc:2238] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.710013 2024 raft_consensus.cc:2267] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.710884 2024 ts_tablet_manager.cc:1905] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.707821 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab4978edea0445fb0456e88e7fb4c44" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:37.716097 854 leader_election.cc:343] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.717455 2215 consensus_queue.cc:1035] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:37.722297 2399 raft_consensus.cc:491] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.722429 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:37.722429 2399 raft_consensus.cc:513] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.722735 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:37.722963 2399 leader_election.cc:290] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:37.725697 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b7c2efd7ea402e8d19038f96b9675c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:37.726178 854 leader_election.cc:343] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.732621 1181 tablet_bootstrap.cc:492] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:37.732754 1181 ts_tablet_manager.cc:1397] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.058s user 0.005s sys 0.004s
I20250112 02:52:37.733476 1181 raft_consensus.cc:357] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.733644 1181 raft_consensus.cc:383] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.733692 1181 raft_consensus.cc:738] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.733879 1181 consensus_queue.cc:260] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.734313 1181 ts_tablet_manager.cc:1428] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 02:52:37.734532 1181 tablet_bootstrap.cc:492] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:37.735705 1181 tablet_bootstrap.cc:654] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.736435 1173 tablet_bootstrap.cc:492] T d45fce86e02f46d4a57af718fa2a62c8 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:37.736560 1173 ts_tablet_manager.cc:1397] T d45fce86e02f46d4a57af718fa2a62c8 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.000s
I20250112 02:52:37.737167 1173 raft_consensus.cc:357] T d45fce86e02f46d4a57af718fa2a62c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.737330 1173 raft_consensus.cc:383] T d45fce86e02f46d4a57af718fa2a62c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.737380 1173 raft_consensus.cc:738] T d45fce86e02f46d4a57af718fa2a62c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.737546 1173 consensus_queue.cc:260] T d45fce86e02f46d4a57af718fa2a62c8 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.743256 1173 ts_tablet_manager.cc:1428] T d45fce86e02f46d4a57af718fa2a62c8 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250112 02:52:37.743643 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:37.744084 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:37.749257 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
I20250112 02:52:37.749531 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet f3771166b77e4f90aaa116ab1a9dc299 (table default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6]). Expected version 1 got 0
I20250112 02:52:37.749812 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:37.750828 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet f924459d1150472aafb325a11825db23 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]). Expected version 3 got 0
I20250112 02:52:37.752442 2024 ts_tablet_manager.cc:1918] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.752776 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet fccff64af8ca4400afc116db98918119 (table default.Table_15bcb2c0f2a74458a1bed231fd3437fc [id=ca9c8281bf254f248076cb600e7c4f87]). Expected version 2 got 0
I20250112 02:52:37.753196 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_5e7392cb5a68412f97bddb2677986918 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 1 got 0
I20250112 02:52:37.753553 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:37.753751 2024 log.cc:1198] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/e9fc9a4f1e8944abb76dad6c2ad53d35
I20250112 02:52:37.754161 2024 ts_tablet_manager.cc:1939] T e9fc9a4f1e8944abb76dad6c2ad53d35 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:37.754794 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 6117dc8b32a64535a2f4167a03bed0d2 (table default.Table_365739770a11434dbf260ea912e7ea81 [id=868adb627654450c936ef198e49d853b]). Expected version 1 got 0
I20250112 02:52:37.756323 1177 tablet_bootstrap.cc:492] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:37.756500 1177 ts_tablet_manager.cc:1397] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.121s user 0.002s sys 0.009s
I20250112 02:52:37.757145 1177 raft_consensus.cc:357] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.757366 1177 raft_consensus.cc:383] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.757459 1177 raft_consensus.cc:738] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.757706 1177 consensus_queue.cc:260] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.758153 1177 ts_tablet_manager.cc:1428] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 02:52:37.758185 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:37.758833 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:37.759693 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_5d473cabff2e46e0b531fbce28d9d558 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 1 got 0
I20250112 02:52:37.763502 2003 ts_tablet_manager.cc:1905] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.764509 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:37.764988 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:37.766089 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 1 got 0
I20250112 02:52:37.766759 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:37.767729 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:37.768059 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:37.768306 1181 tablet_bootstrap.cc:492] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:37.768321 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:37.768443 1181 ts_tablet_manager.cc:1397] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.016s
I20250112 02:52:37.768602 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:37.769016 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:37.769009 1181 raft_consensus.cc:357] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.769598 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:37.769788 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
I20250112 02:52:37.769966 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:37.770323 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5cc414d6420c41839621ab858067548b (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]). Expected version 1 got 0
I20250112 02:52:37.770553 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:37.771016 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:37.771277 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:37.772035 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:37.772369 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet f31b777e7e0f43a28839d8c48c6a6590 (table default.Table_5fb592e41cc4406aa27546e7b41fdb88 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 1 got 0
I20250112 02:52:37.772686 452 catalog_manager.cc:5526] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:37.773260 452 catalog_manager.cc:5526] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:37.773759 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:37.774139 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:37.774600 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 13578542e9e741e8817e0ee63b10f1a1
I20250112 02:52:37.774562 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_ad69cfacaf7b4050ab2d3d6ed237e7d0 [id=9e72fad141db43c09c7e4692769b6eba] alter complete (version 1)
I20250112 02:52:37.775172 455 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40610 (request call id 128) took 3009 ms. Trace:
I20250112 02:52:37.775251 455 rpcz_store.cc:276] 0112 02:52:34.765968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:35.728903 (+962935us) service_pool.cc:225] Handling call
0112 02:52:35.729410 (+ 507us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:35.729563 (+ 153us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:52:35.729623 (+ 60us) catalog_manager.cc:2762] Locking tablets
0112 02:52:35.754523 (+ 24900us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:52:36.679567 (+925044us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:52:36.679646 (+ 79us) catalog_manager.cc:2810] Committing in-memory state
0112 02:52:37.774545 (+1094899us) catalog_manager.cc:2814] Aborting table tasks
0112 02:52:37.775122 (+ 577us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:36.427438 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:36.427595 (+ 157us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:36.427604 (+ 9us) write_op.cc:435] Acquired schema lock
0112 02:52:36.427608 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:36.427761 (+ 153us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:36.427774 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:36.427781 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:36.427917 (+ 136us) tablet.cc:698] Row locks acquired
0112 02:52:36.427921 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:36.427962 (+ 41us) write_op.cc:270] Start()
0112 02:52:36.428000 (+ 38us) write_op.cc:276] Timestamp: P: 1736650356427866 usec, L: 0
0112 02:52:36.428006 (+ 6us) op_driver.cc:348] REPLICATION: starting
0112 02:52:36.435893 (+ 7887us) log.cc:844] Serialized 725 byte log entry
0112 02:52:36.436754 (+ 861us) spinlock_profiling.cc:243] Waited 7.7 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 000000000096ceda 00007f3e150e3632 00007f3e1646beb5 00007f3e1646a9d9 00007f3e16472e00 00007f3e16472c0d 0000000000a71ecc 00007f3e098b0b64 00007f3e098b418b 00007f3e098b4008 0000000000a71ecc
0112 02:52:36.678522 (+241768us) op_driver.cc:464] REPLICATION: finished
0112 02:52:36.678822 (+ 300us) write_op.cc:301] APPLY: starting
0112 02:52:36.678834 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:36.678866 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:36.678869 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:36.678919 (+ 50us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:36.678947 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:36.678955 (+ 8us) write_op.cc:312] APPLY: finished
0112 02:52:36.679011 (+ 56us) log.cc:844] Serialized 29 byte log entry
0112 02:52:36.679063 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.679079 (+ 16us) write_op.cc:454] Released schema lock
0112 02:52:36.679084 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:36.679145 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.679149 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":252,"apply.run_cpu_time_us":316,"apply.run_wall_time_us":315,"num_ops":2,"prepare.queue_time_us":672291,"prepare.run_cpu_time_us":1112,"prepare.run_wall_time_us":14028,"raft.queue_time_us":277,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":185,"replication_time_us":250490,"spinlock_wait_cycles":17699712}]]}
I20250112 02:52:37.779947 1181 raft_consensus.cc:383] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.780041 1181 raft_consensus.cc:738] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.780112 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 13578542e9e741e8817e0ee63b10f1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:37.780225 1181 consensus_queue.cc:260] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.780675 1181 ts_tablet_manager.cc:1428] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250112 02:52:37.780897 1181 tablet_bootstrap.cc:492] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:37.781692 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 13578542e9e741e8817e0ee63b10f1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:37.782253 1181 tablet_bootstrap.cc:654] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.786211 1173 tablet_bootstrap.cc:492] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:37.787689 1173 tablet_bootstrap.cc:654] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.788270 2381 log.cc:826] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Log is configured to *not* fsync() on all Append() calls
I20250112 02:52:37.789326 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 68) took 3274 ms. Trace:
I20250112 02:52:37.789397 2024 rpcz_store.cc:276] 0112 02:52:34.514675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.515166 (+ 491us) service_pool.cc:225] Handling call
0112 02:52:37.705834 (+3190668us) ts_tablet_manager.cc:1145] Deleting tablet e9fc9a4f1e8944abb76dad6c2ad53d35
0112 02:52:37.706414 (+ 580us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.720560 (+ 14146us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.752328 (+ 31768us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.789309 (+ 36981us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3166007}
I20250112 02:52:37.789688 2024 tablet_replica.cc:331] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:37.789842 2024 raft_consensus.cc:2238] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.789963 2024 raft_consensus.cc:2267] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.790374 2024 ts_tablet_manager.cc:1905] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.793566 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet e9fc9a4f1e8944abb76dad6c2ad53d35 (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]) successfully deleted
I20250112 02:52:37.796738 2003 ts_tablet_manager.cc:1918] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.796862 2003 log.cc:1198] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/352c87f1dc724e7b94bc2e64abfce8d7
I20250112 02:52:37.797200 2003 ts_tablet_manager.cc:1939] T 352c87f1dc724e7b94bc2e64abfce8d7 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:37.799014 2024 ts_tablet_manager.cc:1918] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.800937 2399 raft_consensus.cc:491] T a437952889494b0e820e41fdfffe708d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.801102 2399 raft_consensus.cc:513] T a437952889494b0e820e41fdfffe708d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.801769 2024 log.cc:1198] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/04f135b226774285a077e5806131d91b
I20250112 02:52:37.801888 2399 leader_election.cc:290] T a437952889494b0e820e41fdfffe708d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:37.802312 2024 ts_tablet_manager.cc:1939] T 04f135b226774285a077e5806131d91b P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:37.802886 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 90) took 3267 ms. Trace:
I20250112 02:52:37.802783 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a437952889494b0e820e41fdfffe708d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:37.803818 2024 rpcz_store.cc:276] 0112 02:52:34.535037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.539834 (+ 4797us) service_pool.cc:225] Handling call
0112 02:52:37.789646 (+3249812us) ts_tablet_manager.cc:1145] Deleting tablet 04f135b226774285a077e5806131d91b
0112 02:52:37.789654 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.796112 (+ 6458us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.798950 (+ 2838us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.802873 (+ 3923us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3249680}
W20250112 02:52:37.804008 854 leader_election.cc:343] T a437952889494b0e820e41fdfffe708d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.805104 2317 raft_consensus.cc:491] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.805277 2317 raft_consensus.cc:513] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.806118 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0c3191973a477e949fbf9dfcca963f" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:37.806303 910 raft_consensus.cc:2463] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:37.806720 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 04f135b226774285a077e5806131d91b (table default.Table_d1e5cef5fa3e4d90bada176f80bd4180 [id=f93b5bb22ec1424ea047383ed9713618]) successfully deleted
I20250112 02:52:37.806743 713 leader_election.cc:304] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:37.807281 2445 raft_consensus.cc:2798] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:37.807381 2445 raft_consensus.cc:491] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:37.807451 2445 raft_consensus.cc:3054] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.808321 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0c3191973a477e949fbf9dfcca963f" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:37.808486 1049 raft_consensus.cc:2463] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:37.809751 2024 tablet_replica.cc:331] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:37.810091 2024 raft_consensus.cc:2238] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:37.810160 2317 leader_election.cc:290] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:37.810343 826 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:37.811560 2024 raft_consensus.cc:2267] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.812762 2024 ts_tablet_manager.cc:1905] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.823215 2024 ts_tablet_manager.cc:1918] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.824344 2445 raft_consensus.cc:513] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.827495 2399 raft_consensus.cc:491] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.827661 2399 raft_consensus.cc:513] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.828297 2399 leader_election.cc:290] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:37.832221 1181 tablet_bootstrap.cc:492] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:37.832341 1181 ts_tablet_manager.cc:1397] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.004s
I20250112 02:52:37.832597 1177 tablet_bootstrap.cc:492] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:37.833002 1181 raft_consensus.cc:357] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.833164 1181 raft_consensus.cc:383] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.833209 1181 raft_consensus.cc:738] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.833372 1181 consensus_queue.cc:260] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.833825 1181 ts_tablet_manager.cc:1428] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:37.833917 1177 tablet_bootstrap.cc:654] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.834049 1181 tablet_bootstrap.cc:492] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:37.834666 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 79) took 3575 ms. Trace:
I20250112 02:52:37.834744 2003 rpcz_store.cc:276] 0112 02:52:34.259448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.259680 (+ 232us) service_pool.cc:225] Handling call
0112 02:52:37.622120 (+3362440us) ts_tablet_manager.cc:1145] Deleting tablet 352c87f1dc724e7b94bc2e64abfce8d7
0112 02:52:37.622127 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.766071 (+143944us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.796649 (+ 30578us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.834651 (+ 38002us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3362288}
I20250112 02:52:37.835155 2003 tablet_replica.cc:331] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:37.835139 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 352c87f1dc724e7b94bc2e64abfce8d7 (table default.Table_83212f303f0b4413aa6c62887da9dd34 [id=399c12e9075d4e55b21b17837ead62da]) successfully deleted
I20250112 02:52:37.835332 2003 raft_consensus.cc:2238] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:37.835739 1181 tablet_bootstrap.cc:654] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.835638 2003 raft_consensus.cc:2267] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.836447 2003 ts_tablet_manager.cc:1905] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.838148 1181 tablet_bootstrap.cc:492] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:37.838268 1181 ts_tablet_manager.cc:1397] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20250112 02:52:37.838861 1181 raft_consensus.cc:357] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.839023 1181 raft_consensus.cc:383] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.839078 1181 raft_consensus.cc:738] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.839243 1181 consensus_queue.cc:260] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.825371 1173 tablet_bootstrap.cc:492] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:37.828917 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d1beac4410422bbf1e912d35999f3d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:37.839515 2024 log.cc:1198] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/87759e8b99b04e628badc8f60cb38bb3
I20250112 02:52:37.839586 1173 ts_tablet_manager.cc:1397] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.053s user 0.009s sys 0.000s
I20250112 02:52:37.839908 2024 ts_tablet_manager.cc:1939] T 87759e8b99b04e628badc8f60cb38bb3 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
W20250112 02:52:37.839994 854 leader_election.cc:343] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.840281 1173 raft_consensus.cc:357] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.840444 1173 raft_consensus.cc:383] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.840502 1173 raft_consensus.cc:738] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.840708 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 69) took 3324 ms. Trace:
I20250112 02:52:37.840682 1173 consensus_queue.cc:260] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.840822 2024 rpcz_store.cc:276] 0112 02:52:34.516776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.516896 (+ 120us) service_pool.cc:225] Handling call
0112 02:52:37.809711 (+3292815us) ts_tablet_manager.cc:1145] Deleting tablet 87759e8b99b04e628badc8f60cb38bb3
0112 02:52:37.809720 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.819976 (+ 10256us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.823130 (+ 3154us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.840693 (+ 17563us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3266046}
I20250112 02:52:37.846041 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 87759e8b99b04e628badc8f60cb38bb3 (table default.Table_abf2c7b9b8da40c6b9a8381b4e41bda6 [id=4a81fc17cbb94a23a53b5430b545a756]) successfully deleted
I20250112 02:52:37.841166 1173 ts_tablet_manager.cc:1428] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 02:52:37.849232 1181 ts_tablet_manager.cc:1428] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250112 02:52:37.849392 2024 tablet_replica.cc:331] T 1c639a2bcd4740e4be7b2b162004669a P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:37.849563 1173 tablet_bootstrap.cc:492] T 223eb8b4d1024985844867d5641600d1 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:37.849586 2024 raft_consensus.cc:2238] T 1c639a2bcd4740e4be7b2b162004669a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.849798 2024 raft_consensus.cc:2267] T 1c639a2bcd4740e4be7b2b162004669a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.850545 2024 ts_tablet_manager.cc:1905] T 1c639a2bcd4740e4be7b2b162004669a P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.853571 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0c3191973a477e949fbf9dfcca963f" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:37.853793 912 raft_consensus.cc:3054] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.856187 1177 tablet_bootstrap.cc:492] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:37.856313 1177 ts_tablet_manager.cc:1397] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.003s
I20250112 02:52:37.856562 1173 tablet_bootstrap.cc:654] T 223eb8b4d1024985844867d5641600d1 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.856945 1177 raft_consensus.cc:357] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.857105 1177 raft_consensus.cc:383] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.857165 1177 raft_consensus.cc:738] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.857424 1181 tablet_bootstrap.cc:492] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:37.857365 1177 consensus_queue.cc:260] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.857892 1177 ts_tablet_manager.cc:1428] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:37.858268 1177 tablet_bootstrap.cc:492] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:37.858732 1181 tablet_bootstrap.cc:654] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.859488 1177 tablet_bootstrap.cc:654] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.860509 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0c3191973a477e949fbf9dfcca963f" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:37.860656 1049 raft_consensus.cc:3054] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.847926 2003 ts_tablet_manager.cc:1918] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.861594 2003 log.cc:1198] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/a85300a9aa1c4c50a3957a004bea7933
I20250112 02:52:37.861914 2003 ts_tablet_manager.cc:1939] T a85300a9aa1c4c50a3957a004bea7933 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:37.863327 1049 raft_consensus.cc:2463] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:37.864001 714 leader_election.cc:304] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:37.864212 2047 raft_consensus.cc:2798] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:37.864272 2445 leader_election.cc:290] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:37.864315 2047 raft_consensus.cc:695] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:37.857264 912 raft_consensus.cc:2463] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:37.864527 2047 consensus_queue.cc:237] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.867269 2024 ts_tablet_manager.cc:1918] T 1c639a2bcd4740e4be7b2b162004669a P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.868670 1181 tablet_bootstrap.cc:492] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:37.868798 1181 ts_tablet_manager.cc:1397] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20250112 02:52:37.877003 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 78) took 3617 ms. Trace:
I20250112 02:52:37.877055 2024 log.cc:1198] T 1c639a2bcd4740e4be7b2b162004669a P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/1c639a2bcd4740e4be7b2b162004669a
I20250112 02:52:37.877087 2003 rpcz_store.cc:276] 0112 02:52:34.259304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.259900 (+ 596us) service_pool.cc:225] Handling call
0112 02:52:37.835116 (+3575216us) ts_tablet_manager.cc:1145] Deleting tablet a85300a9aa1c4c50a3957a004bea7933
0112 02:52:37.835124 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.843884 (+ 8760us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.847858 (+ 3974us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.876986 (+ 29128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3574426}
I20250112 02:52:37.877452 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet a85300a9aa1c4c50a3957a004bea7933 (table default.Table_679c07c6f25644c6a4136aa56dd5100a [id=3a4127ef6ed145d4bdbb562adadfbb5c]) successfully deleted
I20250112 02:52:37.877526 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 52) took 3652 ms. Trace:
I20250112 02:52:37.877552 2024 ts_tablet_manager.cc:1939] T 1c639a2bcd4740e4be7b2b162004669a P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:37.877588 2003 rpcz_store.cc:276] 0112 02:52:34.225197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.225304 (+ 107us) service_pool.cc:225] Handling call
0112 02:52:37.877428 (+3652124us) ts_tablet_manager.cc:1145] Deleting tablet c4a3486ef011492c8a1df8d6278adedf
0112 02:52:37.877437 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.877513 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3605069}
I20250112 02:52:37.877795 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 70) took 3360 ms. Trace:
I20250112 02:52:37.877893 2003 tablet_replica.cc:331] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
W20250112 02:52:37.877931 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet c4a3486ef011492c8a1df8d6278adedf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4a3486ef011492c8a1df8d6278adedf already in progress: creating tablet
I20250112 02:52:37.877910 2024 rpcz_store.cc:276] 0112 02:52:34.517152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.519112 (+ 1960us) service_pool.cc:225] Handling call
0112 02:52:37.845266 (+3326154us) ts_tablet_manager.cc:1145] Deleting tablet 1c639a2bcd4740e4be7b2b162004669a
0112 02:52:37.849336 (+ 4070us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.853465 (+ 4129us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.867178 (+ 13713us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.877782 (+ 10604us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3304518}
I20250112 02:52:37.878057 2003 raft_consensus.cc:2238] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:37.878217 2024 tablet_replica.cc:331] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:37.878250 2003 pending_rounds.cc:62] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:37.878293 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 1c639a2bcd4740e4be7b2b162004669a (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]) successfully deleted
I20250112 02:52:37.878415 2003 pending_rounds.cc:69] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319861817167872 op_type: NO_OP noop_request { }
I20250112 02:52:37.878458 2024 raft_consensus.cc:2238] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:37.878504 2003 raft_consensus.cc:2883] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:37.878574 2003 raft_consensus.cc:2267] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.878727 2024 raft_consensus.cc:2267] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.879242 2024 ts_tablet_manager.cc:1905] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.879298 2003 ts_tablet_manager.cc:1905] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.881448 1181 raft_consensus.cc:357] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.880175 1173 tablet_bootstrap.cc:492] T 223eb8b4d1024985844867d5641600d1 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:37.881824 1173 ts_tablet_manager.cc:1397] T 223eb8b4d1024985844867d5641600d1 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.005s
I20250112 02:52:37.882452 1173 raft_consensus.cc:357] T 223eb8b4d1024985844867d5641600d1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.882678 1173 raft_consensus.cc:383] T 223eb8b4d1024985844867d5641600d1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.882735 1173 raft_consensus.cc:738] T 223eb8b4d1024985844867d5641600d1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.882913 1173 consensus_queue.cc:260] T 223eb8b4d1024985844867d5641600d1 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.887933 1173 ts_tablet_manager.cc:1428] T 223eb8b4d1024985844867d5641600d1 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20250112 02:52:37.881693 1181 raft_consensus.cc:383] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.890393 1181 raft_consensus.cc:738] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.890614 1181 consensus_queue.cc:260] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.891033 1181 ts_tablet_manager.cc:1428] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20250112 02:52:37.891858 2036 ts_tablet_manager.cc:1918] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.892035 2036 log.cc:1198] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/7f4e4f1e451b48dc9cdd780300f6888c
I20250112 02:52:37.892622 2036 ts_tablet_manager.cc:1939] T 7f4e4f1e451b48dc9cdd780300f6888c P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.892719 2003 ts_tablet_manager.cc:1918] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.892812 2003 log.cc:1198] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/1b184806b4884cd3b0a002905a1e57af
I20250112 02:52:37.893687 1173 tablet_bootstrap.cc:492] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:37.897783 1173 tablet_bootstrap.cc:654] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.898725 1181 tablet_bootstrap.cc:492] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:37.901710 2399 raft_consensus.cc:491] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.901834 2003 ts_tablet_manager.cc:1939] T 1b184806b4884cd3b0a002905a1e57af P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:37.901818 2399 raft_consensus.cc:513] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.902333 2399 leader_election.cc:290] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:37.902503 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 81) took 3642 ms. Trace:
I20250112 02:52:37.902580 2003 rpcz_store.cc:276] 0112 02:52:34.277157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.277247 (+ 90us) service_pool.cc:225] Handling call
0112 02:52:37.877861 (+3600614us) ts_tablet_manager.cc:1145] Deleting tablet 1b184806b4884cd3b0a002905a1e57af
0112 02:52:37.877868 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.882243 (+ 4375us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.892647 (+ 10404us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.902487 (+ 9840us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3596204}
I20250112 02:52:37.903025 2003 tablet_replica.cc:331] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:37.902995 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9c845571f44c8eb8753090d63df45f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:37.903021 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 1b184806b4884cd3b0a002905a1e57af (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]) successfully deleted
I20250112 02:52:37.903179 2003 raft_consensus.cc:2238] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.903306 2003 raft_consensus.cc:2267] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8 [term 2 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:37.903568 854 leader_election.cc:343] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.903756 2003 ts_tablet_manager.cc:1905] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.904285 1181 tablet_bootstrap.cc:654] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.906409 2024 ts_tablet_manager.cc:1918] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.906512 2024 log.cc:1198] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/fb0aba0139cf479b80b32f3dfc936535
I20250112 02:52:37.906855 2024 ts_tablet_manager.cc:1939] T fb0aba0139cf479b80b32f3dfc936535 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:37.908715 2047 raft_consensus.cc:491] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.908849 2047 raft_consensus.cc:513] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.909479 2047 leader_election.cc:290] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:37.909643 2208 raft_consensus.cc:491] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.909849 1177 tablet_bootstrap.cc:492] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:37.909761 2208 raft_consensus.cc:513] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.909955 1177 ts_tablet_manager.cc:1397] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.000s
W20250112 02:52:37.910001 713 leader_election.cc:343] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 416fb6820cd54e118950c68d381f51b2
I20250112 02:52:37.910449 2208 leader_election.cc:290] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:37.910578 1177 raft_consensus.cc:357] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.910748 1177 raft_consensus.cc:383] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.910800 1177 raft_consensus.cc:738] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.910997 1177 consensus_queue.cc:260] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.911293 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416fb6820cd54e118950c68d381f51b2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:37.911530 1177 ts_tablet_manager.cc:1428] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:37.911518 775 raft_consensus.cc:2463] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
W20250112 02:52:37.912077 989 leader_election.cc:343] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 416fb6820cd54e118950c68d381f51b2
I20250112 02:52:37.912220 991 leader_election.cc:304] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:37.912828 2326 raft_consensus.cc:2798] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:37.912899 2326 raft_consensus.cc:491] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:37.912921 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 75) took 3523 ms. Trace:
I20250112 02:52:37.912958 2326 raft_consensus.cc:3054] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.912992 2036 rpcz_store.cc:276] 0112 02:52:34.389699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.389753 (+ 54us) service_pool.cc:225] Handling call
0112 02:52:37.646520 (+3256767us) ts_tablet_manager.cc:1145] Deleting tablet 7f4e4f1e451b48dc9cdd780300f6888c
0112 02:52:37.646528 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.745590 (+ 99062us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.891732 (+146142us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.912903 (+ 21171us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3246345}
I20250112 02:52:37.913324 2036 tablet_replica.cc:331] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.913501 2036 raft_consensus.cc:2238] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.913626 2036 raft_consensus.cc:2267] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.913851 1177 tablet_bootstrap.cc:492] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:37.914050 2036 ts_tablet_manager.cc:1905] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.915025 1177 tablet_bootstrap.cc:654] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.915290 2326 raft_consensus.cc:513] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.916175 2326 leader_election.cc:290] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:37.916689 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416fb6820cd54e118950c68d381f51b2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:37.916842 775 raft_consensus.cc:3054] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.919728 1181 tablet_bootstrap.cc:492] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:37.919834 1181 ts_tablet_manager.cc:1397] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.004s
I20250112 02:52:37.919409 775 raft_consensus.cc:2463] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:37.920519 991 leader_election.cc:304] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:37.920562 1181 raft_consensus.cc:357] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.920727 1181 raft_consensus.cc:383] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.920775 1181 raft_consensus.cc:738] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.920943 2326 raft_consensus.cc:2798] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:37.921193 2326 raft_consensus.cc:695] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:37.921520 2326 consensus_queue.cc:237] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:37.922865 989 leader_election.cc:343] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 416fb6820cd54e118950c68d381f51b2
I20250112 02:52:37.924008 855 leader_election.cc:304] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:37.924811 2399 raft_consensus.cc:2743] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:37.925217 855 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 246550us
I20250112 02:52:37.926007 1173 tablet_bootstrap.cc:492] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:37.926105 1173 ts_tablet_manager.cc:1397] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.002s
I20250112 02:52:37.926360 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab4978edea0445fb0456e88e7fb4c44" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:37.926683 1173 raft_consensus.cc:357] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.926841 1173 raft_consensus.cc:383] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.926883 1173 raft_consensus.cc:738] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.927081 1173 consensus_queue.cc:260] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.927551 1173 ts_tablet_manager.cc:1428] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 02:52:37.927541 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b7c2efd7ea402e8d19038f96b9675c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:37.928622 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a437952889494b0e820e41fdfffe708d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:37.929553 2047 raft_consensus.cc:491] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.929579 1181 consensus_queue.cc:260] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.929688 2047 raft_consensus.cc:513] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.930096 1181 ts_tablet_manager.cc:1428] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250112 02:52:37.930248 2047 leader_election.cc:290] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:37.930348 1181 tablet_bootstrap.cc:492] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:37.929504 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d1beac4410422bbf1e912d35999f3d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:37.930441 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9c845571f44c8eb8753090d63df45f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:37.931016 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 7f4e4f1e451b48dc9cdd780300f6888c (table default.Table_096703f878244b8dac2dcf187fd23fd6 [id=e1779f88b20942eaa2c640ac24946fff]) successfully deleted
W20250112 02:52:37.931419 713 leader_election.cc:343] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: d45fce86e02f46d4a57af718fa2a62c8
I20250112 02:52:37.919426 2003 ts_tablet_manager.cc:1918] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.931877 2003 log.cc:1198] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/d659c5a4f7a446c6aaf2283f65b45106
I20250112 02:52:37.932174 2003 ts_tablet_manager.cc:1939] T d659c5a4f7a446c6aaf2283f65b45106 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:37.932829 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 13578542e9e741e8817e0ee63b10f1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:37.932847 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 63) took 3687 ms. Trace:
I20250112 02:52:37.933064 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416fb6820cd54e118950c68d381f51b2" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:37.933238 1181 tablet_bootstrap.cc:654] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.933625 1177 tablet_bootstrap.cc:492] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:37.933727 1177 ts_tablet_manager.cc:1397] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
W20250112 02:52:37.933542 855 leader_election.cc:343] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.934027 855 leader_election.cc:304] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:37.934291 1177 raft_consensus.cc:357] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.934485 1177 raft_consensus.cc:383] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.934542 1177 raft_consensus.cc:738] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.934628 2399 raft_consensus.cc:2743] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:37.934739 1177 consensus_queue.cc:260] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.935149 1177 ts_tablet_manager.cc:1428] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 02:52:37.935250 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45fce86e02f46d4a57af718fa2a62c8" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:37.935420 1051 raft_consensus.cc:2463] T d45fce86e02f46d4a57af718fa2a62c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:37.936018 827 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:37.936118 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 71) took 3416 ms. Trace:
I20250112 02:52:37.936225 2024 rpcz_store.cc:276] 0112 02:52:34.519413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.519824 (+ 411us) service_pool.cc:225] Handling call
0112 02:52:37.878178 (+3358354us) ts_tablet_manager.cc:1145] Deleting tablet fb0aba0139cf479b80b32f3dfc936535
0112 02:52:37.878187 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.897882 (+ 19695us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.906314 (+ 8432us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.936105 (+ 29791us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3337314}
I20250112 02:52:37.936558 2024 tablet_replica.cc:331] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:37.936695 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet fb0aba0139cf479b80b32f3dfc936535 (table default.Table_672d7452ad6d4c99ae9f2ae77b592f78 [id=33dad7ab4106404aaf33266ed1ea5635]) successfully deleted
I20250112 02:52:37.936875 2024 raft_consensus.cc:2238] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.937049 2024 raft_consensus.cc:2267] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.937043 714 leader_election.cc:304] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:37.937259 2047 raft_consensus.cc:2743] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:37.937427 714 leader_election.cc:304] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
W20250112 02:52:37.937288 855 leader_election.cc:343] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.937644 2047 raft_consensus.cc:2798] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:37.937703 2047 raft_consensus.cc:491] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:37.937767 2047 raft_consensus.cc:3054] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.938267 1173 tablet_bootstrap.cc:492] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:37.938086 855 leader_election.cc:304] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:37.939687 1173 tablet_bootstrap.cc:654] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.939713 2399 raft_consensus.cc:2743] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:37.940027 2047 raft_consensus.cc:513] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.940677 2047 leader_election.cc:290] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:37.941210 713 leader_election.cc:343] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: d45fce86e02f46d4a57af718fa2a62c8
I20250112 02:52:37.941833 2024 ts_tablet_manager.cc:1905] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.942420 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:37.942248 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45fce86e02f46d4a57af718fa2a62c8" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:37.942685 1051 raft_consensus.cc:3054] T d45fce86e02f46d4a57af718fa2a62c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.942797 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:37.943113 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:37.943356 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:37.943521 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
I20250112 02:52:37.943643 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet f3771166b77e4f90aaa116ab1a9dc299 (table default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6]). Expected version 1 got 0
I20250112 02:52:37.943804 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:37.945247 1051 raft_consensus.cc:2463] T d45fce86e02f46d4a57af718fa2a62c8 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:37.945432 2036 ts_tablet_manager.cc:1918] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:37.945525 2036 log.cc:1198] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/70715db0394d46d9b6b289160b00be7d
I20250112 02:52:37.945796 2036 ts_tablet_manager.cc:1939] T 70715db0394d46d9b6b289160b00be7d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.946106 714 leader_election.cc:304] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:37.946545 2047 raft_consensus.cc:2798] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:37.946730 2047 raft_consensus.cc:695] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:37.947010 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_5e7392cb5a68412f97bddb2677986918 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 1 got 0
I20250112 02:52:37.947225 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:37.947763 1177 tablet_bootstrap.cc:492] T e2f1f1a76e5f4308bf56dd647f02c6d8 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:37.948297 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
W20250112 02:52:37.942051 855 leader_election.cc:343] T a437952889494b0e820e41fdfffe708d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.948554 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:37.948834 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_5d473cabff2e46e0b531fbce28d9d558 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 1 got 0
I20250112 02:52:37.949167 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 1 got 0
I20250112 02:52:37.949199 1177 tablet_bootstrap.cc:654] T e2f1f1a76e5f4308bf56dd647f02c6d8 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.949010 855 leader_election.cc:304] T a437952889494b0e820e41fdfffe708d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:37.946975 2047 consensus_queue.cc:237] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.949859 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:37.950047 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 6117dc8b32a64535a2f4167a03bed0d2 (table default.Table_365739770a11434dbf260ea912e7ea81 [id=868adb627654450c936ef198e49d853b]). Expected version 1 got 0
I20250112 02:52:37.950700 2399 raft_consensus.cc:2743] T a437952889494b0e820e41fdfffe708d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:37.950745 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:37.950979 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:37.932906 2003 rpcz_store.cc:276] 0112 02:52:34.245328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.245369 (+ 41us) service_pool.cc:225] Handling call
0112 02:52:37.902986 (+3657617us) ts_tablet_manager.cc:1145] Deleting tablet d659c5a4f7a446c6aaf2283f65b45106
0112 02:52:37.902993 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.916822 (+ 13829us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.919356 (+ 2534us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.932831 (+ 13475us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3617684}
I20250112 02:52:37.951854 2003 tablet_replica.cc:331] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:37.951861 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet d659c5a4f7a446c6aaf2283f65b45106 (table default.Table_218fd160176b4f55937d334142f2ecbd [id=990e8109058d4dffbbd0ae23fc43783b]) successfully deleted
I20250112 02:52:37.952031 2003 raft_consensus.cc:2238] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.952160 2003 raft_consensus.cc:2267] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.952629 2003 ts_tablet_manager.cc:1905] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:37.952723 855 leader_election.cc:343] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.953588 855 leader_election.cc:304] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:37.954734 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 76) took 3553 ms. Trace:
I20250112 02:52:37.954838 2036 rpcz_store.cc:276] 0112 02:52:34.401220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.401264 (+ 44us) service_pool.cc:225] Handling call
0112 02:52:37.913232 (+3511968us) ts_tablet_manager.cc:1145] Deleting tablet 70715db0394d46d9b6b289160b00be7d
0112 02:52:37.913292 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.931722 (+ 18430us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.945338 (+ 13616us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.954720 (+ 9382us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3511775}
I20250112 02:52:37.955139 2036 tablet_replica.cc:331] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.955325 2036 raft_consensus.cc:2238] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.955518 2036 raft_consensus.cc:2267] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.955791 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:37.956221 2036 ts_tablet_manager.cc:1905] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.959149 2024 ts_tablet_manager.cc:1918] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.959259 2024 log.cc:1198] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/6913b0ac9ceb4dbc9cdf591f159134f7
I20250112 02:52:37.960291 1173 tablet_bootstrap.cc:492] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:37.960418 1173 ts_tablet_manager.cc:1397] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250112 02:52:37.960978 1173 raft_consensus.cc:357] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.961136 1173 raft_consensus.cc:383] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.961191 1173 raft_consensus.cc:738] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.961431 2399 raft_consensus.cc:2743] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:37.961385 1173 consensus_queue.cc:260] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:37.961884 855 leader_election.cc:343] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:37.962040 855 leader_election.cc:304] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:37.962374 2399 raft_consensus.cc:2743] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:37.963013 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 70715db0394d46d9b6b289160b00be7d (table default.Table_bc3feb9849cf4e5db05a15330115dd52 [id=53797be8d53c45f3b805e532889be30c]) successfully deleted
I20250112 02:52:37.963941 1173 ts_tablet_manager.cc:1428] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:37.964097 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:37.964756 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet fccff64af8ca4400afc116db98918119 (table default.Table_15bcb2c0f2a74458a1bed231fd3437fc [id=ca9c8281bf254f248076cb600e7c4f87]). Expected version 2 got 0
I20250112 02:52:37.966907 2036 ts_tablet_manager.cc:1918] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.967192 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet f924459d1150472aafb325a11825db23 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]). Expected version 3 got 0
I20250112 02:52:37.967418 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:37.967701 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:37.968019 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:37.968201 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
I20250112 02:52:37.968402 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:37.968793 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:37.968920 2036 log.cc:1198] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/87759e8b99b04e628badc8f60cb38bb3
I20250112 02:52:37.969311 455 catalog_manager.cc:5526] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:37.969980 2036 ts_tablet_manager.cc:1939] T 87759e8b99b04e628badc8f60cb38bb3 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:37.970326 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5cc414d6420c41839621ab858067548b (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]). Expected version 1 got 0
I20250112 02:52:37.970881 1173 tablet_bootstrap.cc:492] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:37.971761 2326 consensus_queue.cc:1035] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:37.972497 2047 raft_consensus.cc:491] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:37.972631 2047 raft_consensus.cc:513] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.973217 2047 leader_election.cc:290] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:37.973531 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d525dae23744a4adfc2f58cb9945ce" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:37.973726 919 raft_consensus.cc:2463] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:37.973850 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d525dae23744a4adfc2f58cb9945ce" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:37.974102 1051 raft_consensus.cc:2463] T b9d525dae23744a4adfc2f58cb9945ce P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:37.974252 1173 tablet_bootstrap.cc:654] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.974227 713 leader_election.cc:304] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:37.974861 1177 tablet_bootstrap.cc:492] T e2f1f1a76e5f4308bf56dd647f02c6d8 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:37.974833 2047 raft_consensus.cc:2798] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:37.974984 1177 ts_tablet_manager.cc:1397] T e2f1f1a76e5f4308bf56dd647f02c6d8 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20250112 02:52:37.975004 2047 raft_consensus.cc:491] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:37.975147 2047 raft_consensus.cc:3054] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.975605 1177 raft_consensus.cc:357] T e2f1f1a76e5f4308bf56dd647f02c6d8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.975778 1177 raft_consensus.cc:383] T e2f1f1a76e5f4308bf56dd647f02c6d8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.975833 1177 raft_consensus.cc:738] T e2f1f1a76e5f4308bf56dd647f02c6d8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.976004 1177 consensus_queue.cc:260] T e2f1f1a76e5f4308bf56dd647f02c6d8 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:37.977505 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 77) took 3573 ms. Trace:
I20250112 02:52:37.977584 2036 rpcz_store.cc:276] 0112 02:52:34.403712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.403777 (+ 65us) service_pool.cc:225] Handling call
0112 02:52:37.955103 (+3551326us) ts_tablet_manager.cc:1145] Deleting tablet 87759e8b99b04e628badc8f60cb38bb3
0112 02:52:37.955112 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.958867 (+ 3755us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.966847 (+ 7980us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.977490 (+ 10643us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6400,"tablet-delete.queue_time_us":3551188}
I20250112 02:52:37.977931 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 87759e8b99b04e628badc8f60cb38bb3 (table default.Table_abf2c7b9b8da40c6b9a8381b4e41bda6 [id=4a81fc17cbb94a23a53b5430b545a756]) successfully deleted
I20250112 02:52:37.978413 2047 raft_consensus.cc:513] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.972467 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:37.978804 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:37.979238 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d525dae23744a4adfc2f58cb9945ce" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:37.979364 1051 raft_consensus.cc:3054] T b9d525dae23744a4adfc2f58cb9945ce P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.979753 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet f31b777e7e0f43a28839d8c48c6a6590 (table default.Table_5fb592e41cc4406aa27546e7b41fdb88 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 1 got 0
I20250112 02:52:37.979840 2047 leader_election.cc:290] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:37.979951 2036 tablet_replica.cc:331] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:37.980145 2036 raft_consensus.cc:2238] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.980306 2036 raft_consensus.cc:2267] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.980458 1177 ts_tablet_manager.cc:1428] T e2f1f1a76e5f4308bf56dd647f02c6d8 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 02:52:37.980582 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d525dae23744a4adfc2f58cb9945ce" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:37.980778 919 raft_consensus.cc:3054] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:37.981454 1181 tablet_bootstrap.cc:492] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:37.981590 1181 ts_tablet_manager.cc:1397] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.002s
I20250112 02:52:37.981678 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:37.980784 2036 ts_tablet_manager.cc:1905] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.982337 1177 tablet_bootstrap.cc:492] T b8daf4ef3593410eae87672402292be6 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:37.982795 1181 raft_consensus.cc:357] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.982959 1181 raft_consensus.cc:383] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:37.983027 1181 raft_consensus.cc:738] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:37.983184 919 raft_consensus.cc:2463] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:37.983283 1181 consensus_queue.cc:260] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.983641 713 leader_election.cc:304] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:37.983733 1181 ts_tablet_manager.cc:1428] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 02:52:37.983960 1181 tablet_bootstrap.cc:492] T 7ad9c47fcb534619949971796b60eba2 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:37.984208 2326 consensus_queue.cc:1035] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:37.984515 2047 raft_consensus.cc:2798] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:37.984694 2047 raft_consensus.cc:695] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:37.984884 2047 consensus_queue.cc:237] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:37.986212 1177 tablet_bootstrap.cc:654] T b8daf4ef3593410eae87672402292be6 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.987100 1181 tablet_bootstrap.cc:654] T 7ad9c47fcb534619949971796b60eba2 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:37.988718 2003 ts_tablet_manager.cc:1918] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:37.988807 2003 log.cc:1198] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/c7c530aa0aa64e47ae27d61f341d2bd9
I20250112 02:52:37.989099 2003 ts_tablet_manager.cc:1939] T c7c530aa0aa64e47ae27d61f341d2bd9 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:37.995656 2024 ts_tablet_manager.cc:1939] T 6913b0ac9ceb4dbc9cdf591f159134f7 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:37.995797 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 53) took 3756 ms. Trace:
I20250112 02:52:37.995877 2003 rpcz_store.cc:276] 0112 02:52:34.239091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.277531 (+ 38440us) service_pool.cc:225] Handling call
0112 02:52:37.951810 (+3674279us) ts_tablet_manager.cc:1145] Deleting tablet c7c530aa0aa64e47ae27d61f341d2bd9
0112 02:52:37.951821 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.966174 (+ 14353us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.988638 (+ 22464us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.995778 (+ 7140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3662636}
I20250112 02:52:37.996235 2003 tablet_replica.cc:331] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:37.996388 2003 raft_consensus.cc:2238] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:37.996517 2003 raft_consensus.cc:2267] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:37.997373 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 72) took 3477 ms. Trace:
I20250112 02:52:37.997447 2003 ts_tablet_manager.cc:1905] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:37.997561 2024 rpcz_store.cc:276] 0112 02:52:34.520064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.520126 (+ 62us) service_pool.cc:225] Handling call
0112 02:52:37.936519 (+3416393us) ts_tablet_manager.cc:1145] Deleting tablet 6913b0ac9ceb4dbc9cdf591f159134f7
0112 02:52:37.936527 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:37.956649 (+ 20122us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.959087 (+ 2438us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:37.997352 (+ 38265us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3395606}
I20250112 02:52:37.996203 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet c7c530aa0aa64e47ae27d61f341d2bd9 (table default.Table_247a821819994de6adac6fabd06c3a1f [id=60486b9e81254d74b8e08a1cadde13bd]) successfully deleted
I20250112 02:52:37.981868 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:37.998567 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:38.001664 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:38.003985 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet a3ffdc237f8640f0b378809f93fc6292
I20250112 02:52:38.004563 450 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40610 (request call id 130) took 3192 ms. Trace:
I20250112 02:52:38.004678 450 rpcz_store.cc:276] 0112 02:52:34.811853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:35.741502 (+929649us) service_pool.cc:225] Handling call
0112 02:52:35.742515 (+ 1013us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:35.742678 (+ 163us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:52:35.742729 (+ 51us) catalog_manager.cc:2762] Locking tablets
0112 02:52:35.757261 (+ 14532us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:52:36.679683 (+922422us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:52:36.679737 (+ 54us) catalog_manager.cc:2810] Committing in-memory state
0112 02:52:38.003913 (+1324176us) catalog_manager.cc:2814] Aborting table tasks
0112 02:52:38.004522 (+ 609us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:36.441492 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:36.441592 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:36.441597 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:36.441599 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:36.441704 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:36.441712 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:36.441716 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:36.441807 (+ 91us) tablet.cc:698] Row locks acquired
0112 02:52:36.441809 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:36.441836 (+ 27us) write_op.cc:270] Start()
0112 02:52:36.441861 (+ 25us) write_op.cc:276] Timestamp: P: 1736650356441741 usec, L: 0
0112 02:52:36.441865 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:36.441991 (+ 126us) log.cc:844] Serialized 725 byte log entry
0112 02:52:36.678750 (+236759us) op_driver.cc:464] REPLICATION: finished
0112 02:52:36.679206 (+ 456us) write_op.cc:301] APPLY: starting
0112 02:52:36.679216 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:36.679244 (+ 28us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:36.679264 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:36.679314 (+ 50us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:36.679337 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:36.679343 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:36.679388 (+ 45us) log.cc:844] Serialized 29 byte log entry
0112 02:52:36.679422 (+ 34us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.679434 (+ 12us) write_op.cc:454] Released schema lock
0112 02:52:36.679448 (+ 14us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":422,"apply.run_cpu_time_us":310,"apply.run_wall_time_us":309,"num_ops":2,"prepare.queue_time_us":683632,"prepare.run_cpu_time_us":606,"prepare.run_wall_time_us":20052,"raft.queue_time_us":303,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":210,"replication_time_us":236848}]]}
I20250112 02:52:38.001129 1051 raft_consensus.cc:2463] T b9d525dae23744a4adfc2f58cb9945ce P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:38.006865 2024 tablet_replica.cc:331] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:38.007025 2024 raft_consensus.cc:2238] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:38.007261 2024 raft_consensus.cc:2267] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.007828 2024 ts_tablet_manager.cc:1905] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.011183 1173 tablet_bootstrap.cc:492] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:38.011287 1173 ts_tablet_manager.cc:1397] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.004s
I20250112 02:52:38.012012 1173 raft_consensus.cc:357] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.012147 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ffdc237f8640f0b378809f93fc6292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.012200 1173 raft_consensus.cc:383] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.012253 1173 raft_consensus.cc:738] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.012363 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 6913b0ac9ceb4dbc9cdf591f159134f7 (table default.Table_f4d612cd6a524bc493147493c5a4a284 [id=4ac16bc62bbe4ed7a977d22c92ad580d]) successfully deleted
I20250112 02:52:38.012442 1173 consensus_queue.cc:260] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.012583 2399 raft_consensus.cc:491] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.012722 2399 raft_consensus.cc:513] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.013047 1173 ts_tablet_manager.cc:1428] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:38.013317 2399 leader_election.cc:290] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:38.014701 742 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ffdc237f8640f0b378809f93fc6292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:38.015507 989 consensus_peers.cc:487] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4abf59d96414e5faa7cf75ba133832c. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:38.015615 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73f2fa452b84be398476b8aba6c9a08" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:38.015661 1173 tablet_bootstrap.cc:492] T 856fa83dedfd4b049e3e0d91d7ac5e40 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:38.016146 2326 consensus_queue.cc:1035] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:52:38.016885 854 leader_election.cc:343] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.017145 1173 tablet_bootstrap.cc:654] T 856fa83dedfd4b049e3e0d91d7ac5e40 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.017657 889 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ffdc237f8640f0b378809f93fc6292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.017858 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73f2fa452b84be398476b8aba6c9a08" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:38.018151 2445 raft_consensus.cc:491] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.018283 2445 raft_consensus.cc:513] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:38.018388 855 leader_election.cc:343] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.018505 855 leader_election.cc:304] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:38.018759 2445 leader_election.cc:290] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.018888 2399 raft_consensus.cc:2743] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:38.019057 2003 ts_tablet_manager.cc:1918] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:38.019161 2003 log.cc:1198] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/6913b0ac9ceb4dbc9cdf591f159134f7
I20250112 02:52:38.019215 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223eb8b4d1024985844867d5641600d1" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:38.019366 919 raft_consensus.cc:2463] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:38.019824 713 leader_election.cc:304] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:38.020671 2445 raft_consensus.cc:2798] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:38.020776 2445 raft_consensus.cc:491] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:38.020871 2445 raft_consensus.cc:3054] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.021216 2003 ts_tablet_manager.cc:1939] T 6913b0ac9ceb4dbc9cdf591f159134f7 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:38.021507 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 66) took 3775 ms. Trace:
I20250112 02:52:38.021588 2003 rpcz_store.cc:276] 0112 02:52:34.246401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.246442 (+ 41us) service_pool.cc:225] Handling call
0112 02:52:37.996200 (+3749758us) ts_tablet_manager.cc:1145] Deleting tablet 6913b0ac9ceb4dbc9cdf591f159134f7
0112 02:52:37.996208 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.007727 (+ 11519us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.018987 (+ 11260us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.021468 (+ 2481us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3706877}
I20250112 02:52:38.021943 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 6913b0ac9ceb4dbc9cdf591f159134f7 (table default.Table_f4d612cd6a524bc493147493c5a4a284 [id=4ac16bc62bbe4ed7a977d22c92ad580d]) successfully deleted
I20250112 02:52:38.023253 2445 raft_consensus.cc:513] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.023702 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223eb8b4d1024985844867d5641600d1" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:38.023845 1051 raft_consensus.cc:2463] T 223eb8b4d1024985844867d5641600d1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:38.024876 1177 tablet_bootstrap.cc:492] T b8daf4ef3593410eae87672402292be6 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:38.024895 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223eb8b4d1024985844867d5641600d1" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:38.025012 1051 raft_consensus.cc:3054] T 223eb8b4d1024985844867d5641600d1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.025116 1177 ts_tablet_manager.cc:1397] T b8daf4ef3593410eae87672402292be6 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.000s
I20250112 02:52:38.026011 1177 raft_consensus.cc:357] T b8daf4ef3593410eae87672402292be6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.026732 1177 raft_consensus.cc:383] T b8daf4ef3593410eae87672402292be6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.026851 1177 raft_consensus.cc:738] T b8daf4ef3593410eae87672402292be6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.027137 1051 raft_consensus.cc:2463] T 223eb8b4d1024985844867d5641600d1 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:38.027151 1177 consensus_queue.cc:260] T b8daf4ef3593410eae87672402292be6 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.027570 714 leader_election.cc:304] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:38.027844 2317 raft_consensus.cc:2798] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:38.027971 2317 raft_consensus.cc:695] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:38.028195 2317 consensus_queue.cc:237] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.031019 2179 raft_consensus.cc:491] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.031057 2445 leader_election.cc:290] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.031144 2179 raft_consensus.cc:513] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.031484 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223eb8b4d1024985844867d5641600d1" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:38.031648 2179 leader_election.cc:290] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.031697 919 raft_consensus.cc:3054] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.032096 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f234256b343f447dadfc3d1936a824ee" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:38.032322 1051 raft_consensus.cc:2463] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:38.032769 714 leader_election.cc:304] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:38.033056 2179 raft_consensus.cc:2798] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:38.033144 2179 raft_consensus.cc:491] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:38.033226 2179 raft_consensus.cc:3054] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.034096 919 raft_consensus.cc:2463] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
W20250112 02:52:38.035768 713 leader_election.cc:343] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: f234256b343f447dadfc3d1936a824ee
I20250112 02:52:38.035825 2179 raft_consensus.cc:513] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.036348 2179 leader_election.cc:290] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.036850 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f234256b343f447dadfc3d1936a824ee" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:38.036979 1049 raft_consensus.cc:3054] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:38.037019 713 leader_election.cc:343] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: f234256b343f447dadfc3d1936a824ee
I20250112 02:52:38.037261 1177 ts_tablet_manager.cc:1428] T b8daf4ef3593410eae87672402292be6 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20250112 02:52:38.038211 2003 tablet_replica.cc:331] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:38.038692 1177 tablet_bootstrap.cc:492] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:38.039165 1049 raft_consensus.cc:2463] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:38.039284 2003 raft_consensus.cc:2238] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.040218 2003 raft_consensus.cc:2267] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.044014 714 leader_election.cc:304] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:38.059906 2326 raft_consensus.cc:491] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.060027 2326 raft_consensus.cc:513] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.060642 2326 leader_election.cc:290] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:38.063699 989 leader_election.cc:343] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: fe614eaeb793473d9b2da206b7345562
I20250112 02:52:38.064085 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe614eaeb793473d9b2da206b7345562" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:38.064539 2003 ts_tablet_manager.cc:1905] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.065727 778 raft_consensus.cc:2463] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:38.066119 991 leader_election.cc:304] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:38.066208 2317 raft_consensus.cc:2798] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:38.066570 2326 raft_consensus.cc:2798] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:38.066646 2326 raft_consensus.cc:491] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:38.066705 2326 raft_consensus.cc:3054] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.066876 2317 raft_consensus.cc:695] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:38.067849 2317 consensus_queue.cc:237] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.069470 2326 raft_consensus.cc:513] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.069932 1177 tablet_bootstrap.cc:654] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.070736 2326 leader_election.cc:290] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:38.071213 989 leader_election.cc:343] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: fe614eaeb793473d9b2da206b7345562
I20250112 02:52:38.071305 2024 ts_tablet_manager.cc:1918] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:38.071395 2024 log.cc:1198] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/7a18132549374218893f35a4e4226b72
I20250112 02:52:38.071586 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe614eaeb793473d9b2da206b7345562" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:38.071897 775 raft_consensus.cc:3054] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.075448 775 raft_consensus.cc:2463] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:38.075850 991 leader_election.cc:304] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:38.076366 2399 raft_consensus.cc:491] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.076534 2399 raft_consensus.cc:513] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.077129 2399 leader_election.cc:290] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:38.078076 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1d9fae290ec4281839991f9d40b105c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:38.078598 855 leader_election.cc:343] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.078969 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1d9fae290ec4281839991f9d40b105c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:38.079578 854 leader_election.cc:343] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.079711 854 leader_election.cc:304] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:38.080085 2399 raft_consensus.cc:2743] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:38.080451 2326 raft_consensus.cc:2798] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:38.080806 2326 raft_consensus.cc:695] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:38.081115 2326 consensus_queue.cc:237] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.084184 1173 tablet_bootstrap.cc:492] T 856fa83dedfd4b049e3e0d91d7ac5e40 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:38.084415 1173 ts_tablet_manager.cc:1397] T 856fa83dedfd4b049e3e0d91d7ac5e40 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.069s user 0.002s sys 0.008s
I20250112 02:52:38.085517 1173 raft_consensus.cc:357] T 856fa83dedfd4b049e3e0d91d7ac5e40 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.085762 1173 raft_consensus.cc:383] T 856fa83dedfd4b049e3e0d91d7ac5e40 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.085894 1173 raft_consensus.cc:738] T 856fa83dedfd4b049e3e0d91d7ac5e40 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.086216 1173 consensus_queue.cc:260] T 856fa83dedfd4b049e3e0d91d7ac5e40 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.088050 2024 ts_tablet_manager.cc:1939] T 7a18132549374218893f35a4e4226b72 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:38.092914 2036 ts_tablet_manager.cc:1918] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:38.093039 2036 log.cc:1198] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/1c639a2bcd4740e4be7b2b162004669a
I20250112 02:52:38.093555 2036 ts_tablet_manager.cc:1939] T 1c639a2bcd4740e4be7b2b162004669a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:38.096906 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 78) took 3692 ms. Trace:
I20250112 02:52:38.096994 2036 rpcz_store.cc:276] 0112 02:52:34.404038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.404073 (+ 35us) service_pool.cc:225] Handling call
0112 02:52:37.977807 (+3573734us) ts_tablet_manager.cc:1145] Deleting tablet 1c639a2bcd4740e4be7b2b162004669a
0112 02:52:37.977816 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.001063 (+ 23247us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.092820 (+ 91757us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.096877 (+ 4057us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3573595}
I20250112 02:52:38.097366 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 1c639a2bcd4740e4be7b2b162004669a (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]) successfully deleted
I20250112 02:52:38.098019 2036 tablet_replica.cc:331] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:38.098184 2036 raft_consensus.cc:2238] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.098307 2036 raft_consensus.cc:2267] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.098795 2036 ts_tablet_manager.cc:1905] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.101154 2003 ts_tablet_manager.cc:1918] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:38.101364 2003 log.cc:1198] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/73136be25c79491784d3ea71aa9b27f2
I20250112 02:52:38.101984 2003 ts_tablet_manager.cc:1939] T 73136be25c79491784d3ea71aa9b27f2 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:38.104563 2036 ts_tablet_manager.cc:1918] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:38.104672 2036 log.cc:1198] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/f7cd0927c6a4401cbe69f55d4db9963a
I20250112 02:52:38.104925 2036 ts_tablet_manager.cc:1939] T f7cd0927c6a4401cbe69f55d4db9963a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:38.105702 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 79) took 3700 ms. Trace:
I20250112 02:52:38.105800 2036 rpcz_store.cc:276] 0112 02:52:34.404308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.404341 (+ 33us) service_pool.cc:225] Handling call
0112 02:52:38.097965 (+3693624us) ts_tablet_manager.cc:1145] Deleting tablet f7cd0927c6a4401cbe69f55d4db9963a
0112 02:52:38.097974 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.101930 (+ 3956us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.104493 (+ 2563us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.105153 (+ 660us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3693514}
I20250112 02:52:38.106220 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet f7cd0927c6a4401cbe69f55d4db9963a (table default.Table_8b4baeb21bec40af9997451f3eceddf1 [id=b7a72bcbde114d5992782721007a0cd4]) successfully deleted
I20250112 02:52:38.107779 2036 tablet_replica.cc:331] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:38.108022 2036 raft_consensus.cc:2238] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.108235 2036 raft_consensus.cc:2267] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.108798 2036 ts_tablet_manager.cc:1905] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.109027 1177 tablet_bootstrap.cc:492] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:38.109292 1177 ts_tablet_manager.cc:1397] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.071s user 0.001s sys 0.008s
I20250112 02:52:38.109486 1173 ts_tablet_manager.cc:1428] T 856fa83dedfd4b049e3e0d91d7ac5e40 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250112 02:52:38.111492 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 82) took 3824 ms. Trace:
I20250112 02:52:38.112058 2003 rpcz_store.cc:276] 0112 02:52:34.290211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.290308 (+ 97us) service_pool.cc:225] Handling call
0112 02:52:38.037849 (+3747541us) ts_tablet_manager.cc:1145] Deleting tablet 73136be25c79491784d3ea71aa9b27f2
0112 02:52:38.037856 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.075304 (+ 37448us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.101015 (+ 25711us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.102558 (+ 1543us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3743558}
I20250112 02:52:38.113229 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 73136be25c79491784d3ea71aa9b27f2 (table default.Table_e3c5ba009c9f4301a3f99d51a8dcf687 [id=d5bf19dc7bff469ba06018568b796a63]) successfully deleted
I20250112 02:52:38.113343 2047 raft_consensus.cc:491] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.113492 2047 raft_consensus.cc:513] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.112731 1177 raft_consensus.cc:357] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.114426 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c10fad790046a88fbc8ebe0303c481" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:38.114634 1049 raft_consensus.cc:2463] T e5c10fad790046a88fbc8ebe0303c481 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:38.114957 2036 ts_tablet_manager.cc:1918] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:38.113801 2003 tablet_replica.cc:331] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:38.115146 2036 log.cc:1198] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/61593969d2234ce3a6089027b067d922
I20250112 02:52:38.113894 1177 raft_consensus.cc:383] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.115612 2036 ts_tablet_manager.cc:1939] T 61593969d2234ce3a6089027b067d922 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:38.115640 1177 raft_consensus.cc:738] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.116073 2003 raft_consensus.cc:2238] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.116436 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 80) took 3711 ms. Trace:
I20250112 02:52:38.114077 2047 leader_election.cc:290] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:38.114718 713 leader_election.cc:343] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: e5c10fad790046a88fbc8ebe0303c481
I20250112 02:52:38.116564 2036 rpcz_store.cc:276] 0112 02:52:34.404569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.404602 (+ 33us) service_pool.cc:225] Handling call
0112 02:52:38.107714 (+3703112us) ts_tablet_manager.cc:1145] Deleting tablet 61593969d2234ce3a6089027b067d922
0112 02:52:38.107724 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.111484 (+ 3760us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.114877 (+ 3393us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.116404 (+ 1527us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3703001}
I20250112 02:52:38.117209 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 61593969d2234ce3a6089027b067d922 (table default.Table_0188036e1bd0455ea82fd355934295fc [id=b2f29883bc3844cf8b008345b849345c]) successfully deleted
I20250112 02:52:38.117334 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 95) took 3581 ms. Trace:
I20250112 02:52:38.117439 2024 rpcz_store.cc:276] 0112 02:52:34.535459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.540955 (+ 5496us) service_pool.cc:225] Handling call
0112 02:52:38.006825 (+3465870us) ts_tablet_manager.cc:1145] Deleting tablet 7a18132549374218893f35a4e4226b72
0112 02:52:38.006834 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.011175 (+ 4341us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.071218 (+ 60043us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.117282 (+ 46064us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3465638}
I20250112 02:52:38.117763 2024 tablet_replica.cc:331] T a85300a9aa1c4c50a3957a004bea7933 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:38.117573 2036 tablet_replica.cc:331] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:38.117846 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 7a18132549374218893f35a4e4226b72 (table default.Table_3fcca85a33054d7ab388f29bba85fe35 [id=18415100e94043e99f9884b853158ae1]) successfully deleted
I20250112 02:52:38.117957 2024 raft_consensus.cc:2238] T a85300a9aa1c4c50a3957a004bea7933 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.118129 2024 raft_consensus.cc:2267] T a85300a9aa1c4c50a3957a004bea7933 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.118140 2036 raft_consensus.cc:2238] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.118369 2036 raft_consensus.cc:2267] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.118660 2024 ts_tablet_manager.cc:1905] T a85300a9aa1c4c50a3957a004bea7933 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.118916 2036 ts_tablet_manager.cc:1905] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.119076 714 leader_election.cc:304] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:38.123026 1173 tablet_bootstrap.cc:492] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:38.124111 1181 tablet_bootstrap.cc:492] T 7ad9c47fcb534619949971796b60eba2 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:38.124277 1181 ts_tablet_manager.cc:1397] T 7ad9c47fcb534619949971796b60eba2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.140s user 0.006s sys 0.003s
I20250112 02:52:38.124562 2399 raft_consensus.cc:491] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.124689 2399 raft_consensus.cc:513] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.125222 2399 leader_election.cc:290] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:38.126047 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea2ef4e9a63491db850bb5393938017" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:38.126514 854 leader_election.cc:343] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.126879 1181 raft_consensus.cc:357] T 7ad9c47fcb534619949971796b60eba2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.127102 1181 raft_consensus.cc:383] T 7ad9c47fcb534619949971796b60eba2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.127197 1181 raft_consensus.cc:738] T 7ad9c47fcb534619949971796b60eba2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.127445 1181 consensus_queue.cc:260] T 7ad9c47fcb534619949971796b60eba2 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.128221 2047 raft_consensus.cc:2798] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:38.128317 2047 raft_consensus.cc:491] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:38.128405 2047 raft_consensus.cc:3054] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.129105 2003 raft_consensus.cc:2267] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.130681 1181 ts_tablet_manager.cc:1428] T 7ad9c47fcb534619949971796b60eba2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20250112 02:52:38.130838 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea2ef4e9a63491db850bb5393938017" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:38.130961 1181 tablet_bootstrap.cc:492] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
W20250112 02:52:38.131353 855 leader_election.cc:343] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.131515 855 leader_election.cc:304] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:38.131913 2399 raft_consensus.cc:2743] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:38.135478 1181 tablet_bootstrap.cc:654] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.136812 2047 raft_consensus.cc:513] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.137421 2047 leader_election.cc:290] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.137586 2036 ts_tablet_manager.cc:1918] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:38.137696 2036 log.cc:1198] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/daef3a58605e40978f7389698b22265d
I20250112 02:52:38.138005 2036 ts_tablet_manager.cc:1939] T daef3a58605e40978f7389698b22265d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:38.138641 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 81) took 3733 ms. Trace:
I20250112 02:52:38.138736 2036 rpcz_store.cc:276] 0112 02:52:34.404816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.404847 (+ 31us) service_pool.cc:225] Handling call
0112 02:52:38.117519 (+3712672us) ts_tablet_manager.cc:1145] Deleting tablet daef3a58605e40978f7389698b22265d
0112 02:52:38.117527 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.121859 (+ 4332us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.136566 (+ 14707us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.138597 (+ 2031us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3712567}
I20250112 02:52:38.139113 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet daef3a58605e40978f7389698b22265d (table default.Table_8e00e020edc14465ad0e935be514d03d [id=cf8528b7c33a47ce984da87382bef6a5]) successfully deleted
W20250112 02:52:38.139714 713 leader_election.cc:343] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: e5c10fad790046a88fbc8ebe0303c481
I20250112 02:52:38.140050 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 84) took 3581 ms. Trace:
I20250112 02:52:38.140157 2036 rpcz_store.cc:276] 0112 02:52:34.558710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.558757 (+ 47us) service_pool.cc:225] Handling call
0112 02:52:38.139919 (+3581162us) ts_tablet_manager.cc:1145] Deleting tablet 62f4bd2df0ce45dabc55373f1154d3ac
0112 02:52:38.139928 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.140007 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3564237}
W20250112 02:52:38.140540 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 62f4bd2df0ce45dabc55373f1154d3ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62f4bd2df0ce45dabc55373f1154d3ac already in progress: creating tablet
I20250112 02:52:38.128988 1177 consensus_queue.cc:260] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.140921 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c10fad790046a88fbc8ebe0303c481" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:38.141278 1049 raft_consensus.cc:3054] T e5c10fad790046a88fbc8ebe0303c481 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.142055 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 86) took 3578 ms. Trace:
I20250112 02:52:38.142194 2036 rpcz_store.cc:276] 0112 02:52:34.563845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.563893 (+ 48us) service_pool.cc:225] Handling call
0112 02:52:38.140841 (+3576948us) ts_tablet_manager.cc:1145] Deleting tablet a8e7882e07784bdda18dc1ffd20963c8
0112 02:52:38.140849 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.142025 (+ 1176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3561279}
W20250112 02:52:38.142649 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet a8e7882e07784bdda18dc1ffd20963c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8e7882e07784bdda18dc1ffd20963c8 already in progress: creating tablet
I20250112 02:52:38.142944 2036 tablet_replica.cc:331] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:38.143225 2036 raft_consensus.cc:2238] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.143400 2036 raft_consensus.cc:2267] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.143762 966 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:38.144426 1049 raft_consensus.cc:2463] T e5c10fad790046a88fbc8ebe0303c481 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:38.144832 714 leader_election.cc:304] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:38.145648 2317 raft_consensus.cc:2798] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:38.145810 2317 raft_consensus.cc:695] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:38.146097 2317 consensus_queue.cc:237] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.144063 2036 ts_tablet_manager.cc:1905] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.145237 2399 raft_consensus.cc:491] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.148386 2399 raft_consensus.cc:513] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.148954 2399 leader_election.cc:290] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:38.151298 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf54f5185a0f4c70b9057309fca14e56" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:38.145368 1181 tablet_bootstrap.cc:492] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:38.151646 1181 ts_tablet_manager.cc:1397] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.002s
W20250112 02:52:38.151859 855 leader_election.cc:343] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.151996 1177 ts_tablet_manager.cc:1428] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.043s user 0.002s sys 0.000s
I20250112 02:52:38.152145 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf54f5185a0f4c70b9057309fca14e56" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:38.152262 2003 ts_tablet_manager.cc:1905] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.152382 1181 raft_consensus.cc:357] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.152577 1181 raft_consensus.cc:383] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.152670 1181 raft_consensus.cc:738] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
W20250112 02:52:38.152772 854 leader_election.cc:343] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.152900 1181 consensus_queue.cc:260] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.153365 1181 ts_tablet_manager.cc:1428] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:38.153802 1177 tablet_bootstrap.cc:492] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:38.155612 2036 ts_tablet_manager.cc:1918] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:38.155725 2036 log.cc:1198] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/8ee98f055024429eafaa29bb961cab87
I20250112 02:52:38.156062 2036 ts_tablet_manager.cc:1939] T 8ee98f055024429eafaa29bb961cab87 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:38.156332 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 87) took 3589 ms. Trace:
I20250112 02:52:38.156414 2036 rpcz_store.cc:276] 0112 02:52:34.567194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.567247 (+ 53us) service_pool.cc:225] Handling call
0112 02:52:38.142893 (+3575646us) ts_tablet_manager.cc:1145] Deleting tablet 8ee98f055024429eafaa29bb961cab87
0112 02:52:38.142902 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.150709 (+ 7807us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.155521 (+ 4812us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.156297 (+ 776us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3563267}
I20250112 02:52:38.156770 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 89) took 3558 ms. Trace:
I20250112 02:52:38.156853 2036 rpcz_store.cc:276] 0112 02:52:34.598333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.599008 (+ 675us) service_pool.cc:225] Handling call
0112 02:52:38.156656 (+3557648us) ts_tablet_manager.cc:1145] Deleting tablet c29a9ca43e034b43a00207e5847bbee3
0112 02:52:38.156664 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.156732 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3557402}
I20250112 02:52:38.157163 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 90) took 3546 ms. Trace:
I20250112 02:52:38.157263 2036 rpcz_store.cc:276] 0112 02:52:34.611032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.611181 (+ 149us) service_pool.cc:225] Handling call
0112 02:52:38.157054 (+3545873us) ts_tablet_manager.cc:1145] Deleting tablet 83c04f1a2cb04950a9f6170e252c19b4
0112 02:52:38.157063 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.157126 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3545700}
I20250112 02:52:38.157595 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 127) took 2418 ms. Trace:
I20250112 02:52:38.157697 2036 rpcz_store.cc:276] 0112 02:52:35.739022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:35.739091 (+ 69us) service_pool.cc:225] Handling call
0112 02:52:38.157485 (+2418394us) ts_tablet_manager.cc:1145] Deleting tablet 82b351c563d749df806ab5d0331a4ef2
0112 02:52:38.157494 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.157562 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2418198}
I20250112 02:52:38.157975 2036 tablet_replica.cc:331] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:38.158157 2036 raft_consensus.cc:2238] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.158284 2036 raft_consensus.cc:2267] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.158702 2036 ts_tablet_manager.cc:1905] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.160195 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 8ee98f055024429eafaa29bb961cab87 (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]) successfully deleted
I20250112 02:52:38.152904 854 leader_election.cc:304] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:38.160424 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet c29a9ca43e034b43a00207e5847bbee3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c29a9ca43e034b43a00207e5847bbee3 already in progress: creating tablet
W20250112 02:52:38.160621 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 83c04f1a2cb04950a9f6170e252c19b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83c04f1a2cb04950a9f6170e252c19b4 already in progress: creating tablet
W20250112 02:52:38.160791 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 82b351c563d749df806ab5d0331a4ef2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82b351c563d749df806ab5d0331a4ef2 already in progress: creating tablet
I20250112 02:52:38.163162 1181 tablet_bootstrap.cc:492] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:38.163941 2326 raft_consensus.cc:491] T 4d25815a5df94e099e7481c48c3da793 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.164507 1181 tablet_bootstrap.cc:654] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.165724 965 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:38.167053 2326 raft_consensus.cc:513] T 4d25815a5df94e099e7481c48c3da793 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.167896 2326 leader_election.cc:290] T 4d25815a5df94e099e7481c48c3da793 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.168366 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d25815a5df94e099e7481c48c3da793" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:38.168744 1177 tablet_bootstrap.cc:654] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.170315 2228 raft_consensus.cc:491] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.170437 2228 raft_consensus.cc:513] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.170907 2228 leader_election.cc:290] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.171525 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2361bc6bd8164c2daa37be6b0845f4bd" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:38.171736 774 raft_consensus.cc:2463] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:38.172477 991 leader_election.cc:304] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:38.172778 2228 raft_consensus.cc:2798] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:38.172899 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:38.173105 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:38.173409 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:38.173417 2228 raft_consensus.cc:491] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:38.173506 2228 raft_consensus.cc:3054] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.173548 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:38.173768 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
I20250112 02:52:38.173913 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet f3771166b77e4f90aaa116ab1a9dc299 (table default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6]). Expected version 1 got 0
I20250112 02:52:38.174113 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:38.175151 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:38.175750 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:38.175834 2399 raft_consensus.cc:2743] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:38.176241 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:38.182919 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
I20250112 02:52:38.183359 2399 raft_consensus.cc:491] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.183521 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 1 got 0
I20250112 02:52:38.183535 2399 raft_consensus.cc:513] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.184113 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:38.184180 2399 leader_election.cc:290] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:38.184545 2036 ts_tablet_manager.cc:1918] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:38.184656 2036 log.cc:1198] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/003a3b4705e943c6a900762ec11ddd47
I20250112 02:52:38.184389 450 catalog_manager.cc:5526] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:38.184700 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37862023095644bab2403576e8cb8c58" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:38.184945 2036 ts_tablet_manager.cc:1939] T 003a3b4705e943c6a900762ec11ddd47 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:38.184953 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 6117dc8b32a64535a2f4167a03bed0d2 (table default.Table_365739770a11434dbf260ea912e7ea81 [id=868adb627654450c936ef198e49d853b]). Expected version 1 got 0
W20250112 02:52:38.185122 855 leader_election.cc:343] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.185217 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 128) took 2440 ms. Trace:
I20250112 02:52:38.185297 2036 rpcz_store.cc:276] 0112 02:52:35.744912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:35.744962 (+ 50us) service_pool.cc:225] Handling call
0112 02:52:38.157904 (+2412942us) ts_tablet_manager.cc:1145] Deleting tablet 003a3b4705e943c6a900762ec11ddd47
0112 02:52:38.157917 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.173110 (+ 15193us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.184463 (+ 11353us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.185188 (+ 725us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2412763}
I20250112 02:52:38.185653 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 003a3b4705e943c6a900762ec11ddd47 (table default.Table_743b3f7029534564b35c0c137b35fd75 [id=b2fa3922f6d447769a8179052f9a1861]) successfully deleted
I20250112 02:52:38.186022 2228 raft_consensus.cc:513] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.186190 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:38.186832 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
W20250112 02:52:38.187357 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 13578542e9e741e8817e0ee63b10f1a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13578542e9e741e8817e0ee63b10f1a1 already in progress: creating tablet
I20250112 02:52:38.187600 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:38.188011 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet fccff64af8ca4400afc116db98918119 (table default.Table_15bcb2c0f2a74458a1bed231fd3437fc [id=ca9c8281bf254f248076cb600e7c4f87]). Expected version 2 got 0
W20250112 02:52:38.188185 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet a3ffdc237f8640f0b378809f93fc6292 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3ffdc237f8640f0b378809f93fc6292 already in progress: creating tablet
I20250112 02:52:38.188328 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet f924459d1150472aafb325a11825db23 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]). Expected version 3 got 0
I20250112 02:52:38.188678 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:38.189021 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:38.189410 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:38.189793 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 5cc414d6420c41839621ab858067548b (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]). Expected version 1 got 0
I20250112 02:52:38.189853 1177 tablet_bootstrap.cc:492] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:38.190009 1177 ts_tablet_manager.cc:1397] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.003s
I20250112 02:52:38.190142 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:38.190383 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:38.190650 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2361bc6bd8164c2daa37be6b0845f4bd" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:38.190861 775 raft_consensus.cc:3054] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.191159 1177 raft_consensus.cc:357] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.191352 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:38.191499 1177 raft_consensus.cc:383] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.191632 1177 raft_consensus.cc:738] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.191628 450 catalog_manager.cc:5526] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:38.192010 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet f31b777e7e0f43a28839d8c48c6a6590 (table default.Table_5fb592e41cc4406aa27546e7b41fdb88 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 1 got 0
I20250112 02:52:38.191991 1177 consensus_queue.cc:260] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.192225 450 catalog_manager.cc:5526] T 1811ce8982fd4003a35e89ea34e32746 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:38.192976 1177 ts_tablet_manager.cc:1428] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 02:52:38.193658 1177 tablet_bootstrap.cc:492] T f05c8426ea3d4fe89fbe6c88c688019f P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:38.195097 2228 leader_election.cc:290] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.195524 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:38.195501 2003 ts_tablet_manager.cc:1918] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:38.195880 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:38.196170 2003 log.cc:1198] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/6cb76746b3a846c3aba531c04bf63966
I20250112 02:52:38.194325 775 raft_consensus.cc:2463] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:38.197669 991 leader_election.cc:304] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:38.198149 2228 raft_consensus.cc:2798] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:38.198349 2228 raft_consensus.cc:695] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:38.198438 2003 ts_tablet_manager.cc:1939] T 6cb76746b3a846c3aba531c04bf63966 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:38.198748 2228 consensus_queue.cc:237] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.199918 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 83) took 3904 ms. Trace:
I20250112 02:52:38.200243 2003 rpcz_store.cc:276] 0112 02:52:34.294945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.294982 (+ 37us) service_pool.cc:225] Handling call
0112 02:52:38.113430 (+3818448us) ts_tablet_manager.cc:1145] Deleting tablet 6cb76746b3a846c3aba531c04bf63966
0112 02:52:38.113439 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.176622 (+ 63183us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.195113 (+ 18491us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.199698 (+ 4585us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3813239}
I20250112 02:52:38.203598 2024 ts_tablet_manager.cc:1918] T a85300a9aa1c4c50a3957a004bea7933 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:38.203706 2024 log.cc:1198] T a85300a9aa1c4c50a3957a004bea7933 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/a85300a9aa1c4c50a3957a004bea7933
I20250112 02:52:38.204022 2024 ts_tablet_manager.cc:1939] T a85300a9aa1c4c50a3957a004bea7933 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:38.204644 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 89) took 3669 ms. Trace:
I20250112 02:52:38.204758 2024 rpcz_store.cc:276] 0112 02:52:34.534952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.539744 (+ 4792us) service_pool.cc:225] Handling call
0112 02:52:38.117682 (+3577938us) ts_tablet_manager.cc:1145] Deleting tablet a85300a9aa1c4c50a3957a004bea7933
0112 02:52:38.117691 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.144240 (+ 26549us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.203491 (+ 59251us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.204609 (+ 1118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3576465}
I20250112 02:52:38.205021 2024 tablet_replica.cc:331] T 6cb76746b3a846c3aba531c04bf63966 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:38.205646 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet a85300a9aa1c4c50a3957a004bea7933 (table default.Table_679c07c6f25644c6a4136aa56dd5100a [id=3a4127ef6ed145d4bdbb562adadfbb5c]) successfully deleted
I20250112 02:52:38.208232 1173 tablet_bootstrap.cc:654] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.211162 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 6cb76746b3a846c3aba531c04bf63966 (table default.Table_75db5f75b2874392b44dd580e0e52de9 [id=013bd2f66fe64ed593d8c0261ffa1c18]) successfully deleted
I20250112 02:52:38.212038 2003 tablet_replica.cc:331] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:38.212814 1177 tablet_bootstrap.cc:654] T f05c8426ea3d4fe89fbe6c88c688019f P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.213583 2047 consensus_queue.cc:1035] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:38.213652 2003 raft_consensus.cc:2238] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.214802 2003 raft_consensus.cc:2267] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.215492 1102 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:38.217423 2024 raft_consensus.cc:2238] T 6cb76746b3a846c3aba531c04bf63966 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.217595 2024 raft_consensus.cc:2267] T 6cb76746b3a846c3aba531c04bf63966 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.217797 2003 ts_tablet_manager.cc:1905] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:38.218112 854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 65332.5us
I20250112 02:52:38.218617 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37862023095644bab2403576e8cb8c58" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:38.218735 2024 ts_tablet_manager.cc:1905] T 6cb76746b3a846c3aba531c04bf63966 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.223835 2047 raft_consensus.cc:491] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.223951 2047 raft_consensus.cc:513] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.224452 2047 leader_election.cc:290] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.224813 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856fa83dedfd4b049e3e0d91d7ac5e40" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:38.224977 912 raft_consensus.cc:2463] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:38.225153 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856fa83dedfd4b049e3e0d91d7ac5e40" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:38.225317 1049 raft_consensus.cc:2463] T 856fa83dedfd4b049e3e0d91d7ac5e40 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:38.225354 713 leader_election.cc:304] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:38.225883 2047 raft_consensus.cc:2798] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:38.226009 2047 raft_consensus.cc:491] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:38.226099 2047 raft_consensus.cc:3054] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.196169 450 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:38.227859 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_9b65dd291c9f4fcdaa6d5e37f04d24ae [id=e63daf724d874c89badafb8d59653f74] alter complete (version 1)
I20250112 02:52:38.228408 2047 raft_consensus.cc:513] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.229329 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856fa83dedfd4b049e3e0d91d7ac5e40" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:38.229480 1053 raft_consensus.cc:3054] T 856fa83dedfd4b049e3e0d91d7ac5e40 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.231433 1053 raft_consensus.cc:2463] T 856fa83dedfd4b049e3e0d91d7ac5e40 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:38.232419 714 leader_election.cc:304] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:38.232870 2047 leader_election.cc:290] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.233067 2047 raft_consensus.cc:2798] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:38.233240 2047 raft_consensus.cc:695] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:38.233542 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856fa83dedfd4b049e3e0d91d7ac5e40" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:38.233572 2047 consensus_queue.cc:237] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.233760 912 raft_consensus.cc:3054] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.234480 2496 raft_consensus.cc:491] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.234663 2496 raft_consensus.cc:513] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.235255 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d25815a5df94e099e7481c48c3da793" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:38.236003 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2361bc6bd8164c2daa37be6b0845f4bd" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:38.236231 774 raft_consensus.cc:2388] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe36951a83fb4e57b26e187edbcbfe3d in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
I20250112 02:52:38.236801 2496 leader_election.cc:290] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:38.238225 912 raft_consensus.cc:2463] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:38.240216 2326 raft_consensus.cc:491] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.240338 2326 raft_consensus.cc:513] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.240984 2326 leader_election.cc:290] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.241964 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2361bc6bd8164c2daa37be6b0845f4bd" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:38.242151 910 raft_consensus.cc:2463] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:38.242458 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8daf4ef3593410eae87672402292be6" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:38.242669 774 raft_consensus.cc:2463] T b8daf4ef3593410eae87672402292be6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:38.244719 778 raft_consensus.cc:3054] T b9537fcd96eb41cc8f680075ec71137b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.246906 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_045e5b104dc64217aa39c22534007cab [id=2c0e94e8c5774da1b8130c7651d94500] alter complete (version 1)
I20250112 02:52:38.247604 2496 consensus_queue.cc:1035] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:38.247959 991 leader_election.cc:304] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:38.248279 2215 raft_consensus.cc:491] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.248406 2215 raft_consensus.cc:513] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.248919 2215 leader_election.cc:290] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.249053 2215 raft_consensus.cc:2798] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:38.249114 2215 raft_consensus.cc:491] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:38.249166 2215 raft_consensus.cc:3054] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.251331 2317 raft_consensus.cc:491] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.251497 2317 raft_consensus.cc:513] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.252137 2317 leader_election.cc:290] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.252605 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_0bdc5f49d4ff4d6fb1290bc2dc0f70d9 [id=e3bb489c53454222a79702b798c0c2f1] alter complete (version 2)
I20250112 02:52:38.252679 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0dc78d722ac4687bfb3dce94d6db7f0" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:38.252833 774 raft_consensus.cc:2463] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:38.252919 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0dc78d722ac4687bfb3dce94d6db7f0" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:38.253118 910 raft_consensus.cc:2463] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:38.253119 991 leader_election.cc:304] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:38.253329 2208 raft_consensus.cc:2798] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:38.253413 2208 raft_consensus.cc:491] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:38.253484 2208 raft_consensus.cc:3054] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.253481 713 leader_election.cc:304] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:38.253746 2317 raft_consensus.cc:2798] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:38.253830 2317 raft_consensus.cc:491] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:38.253911 2317 raft_consensus.cc:3054] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.255741 2208 raft_consensus.cc:513] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.256191 2208 leader_election.cc:290] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.256224 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0dc78d722ac4687bfb3dce94d6db7f0" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:38.256415 1053 raft_consensus.cc:2388] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 289afc116aa74408aa5bff74e11c50d8 in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
I20250112 02:52:38.257045 2317 raft_consensus.cc:513] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.258011 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0dc78d722ac4687bfb3dce94d6db7f0" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:38.258188 1053 raft_consensus.cc:2388] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 289afc116aa74408aa5bff74e11c50d8 in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
I20250112 02:52:38.258664 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0dc78d722ac4687bfb3dce94d6db7f0" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:38.258853 771 raft_consensus.cc:2388] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f1e775296eb746db8b318f17d26820e6 in current term 1: Already voted for candidate 289afc116aa74408aa5bff74e11c50d8 in this term.
I20250112 02:52:38.251407 2215 raft_consensus.cc:513] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.259079 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0dc78d722ac4687bfb3dce94d6db7f0" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:38.259227 910 raft_consensus.cc:3054] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.259275 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_cd1f185d82b3474083246ec3702851fc [id=bfa9a504a8d34c86929efb0d15d049d0] alter complete (version 3)
I20250112 02:52:38.259390 2215 leader_election.cc:290] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.260143 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8daf4ef3593410eae87672402292be6" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:38.260294 771 raft_consensus.cc:3054] T b8daf4ef3593410eae87672402292be6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.261593 910 raft_consensus.cc:2463] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:38.262022 713 leader_election.cc:304] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:38.262108 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2361bc6bd8164c2daa37be6b0845f4bd" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:38.262267 917 raft_consensus.cc:3054] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.262290 2445 raft_consensus.cc:2798] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:38.262336 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet cbe30f9860e2492b90e8cedf8ff9daba
I20250112 02:52:38.262431 2445 raft_consensus.cc:695] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:38.262714 771 raft_consensus.cc:2463] T b8daf4ef3593410eae87672402292be6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:38.262673 2445 consensus_queue.cc:237] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.262864 454 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40610 (request call id 129) took 3472 ms. Trace:
I20250112 02:52:38.262941 454 rpcz_store.cc:276] 0112 02:52:34.790775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:35.737171 (+946396us) service_pool.cc:225] Handling call
0112 02:52:35.737572 (+ 401us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:35.737721 (+ 149us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:52:35.737773 (+ 52us) catalog_manager.cc:2762] Locking tablets
0112 02:52:35.737914 (+ 141us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:52:36.414757 (+676843us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:52:36.414835 (+ 78us) catalog_manager.cc:2810] Committing in-memory state
0112 02:52:38.262293 (+1847458us) catalog_manager.cc:2814] Aborting table tasks
0112 02:52:38.262815 (+ 522us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:36.159035 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:36.159176 (+ 141us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:36.159183 (+ 7us) write_op.cc:435] Acquired schema lock
0112 02:52:36.159186 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:36.159326 (+ 140us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:36.159338 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:36.159343 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:36.215148 (+ 55805us) tablet.cc:698] Row locks acquired
0112 02:52:36.215152 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:36.215196 (+ 44us) write_op.cc:270] Start()
0112 02:52:36.215233 (+ 37us) write_op.cc:276] Timestamp: P: 1736650356215143 usec, L: 0
0112 02:52:36.215237 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:36.215411 (+ 174us) log.cc:844] Serialized 725 byte log entry
0112 02:52:36.413281 (+197870us) op_driver.cc:464] REPLICATION: finished
0112 02:52:36.414280 (+ 999us) write_op.cc:301] APPLY: starting
0112 02:52:36.414299 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:36.414343 (+ 44us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:36.414378 (+ 35us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:36.414468 (+ 90us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:36.414506 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:36.414513 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:36.414581 (+ 68us) log.cc:844] Serialized 29 byte log entry
0112 02:52:36.414644 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.414661 (+ 17us) write_op.cc:454] Released schema lock
0112 02:52:36.414668 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":936,"apply.run_cpu_time_us":464,"apply.run_wall_time_us":463,"num_ops":2,"prepare.queue_time_us":420429,"prepare.run_cpu_time_us":921,"prepare.run_wall_time_us":69519,"raft.queue_time_us":102,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":198013}]]}
I20250112 02:52:38.264847 917 raft_consensus.cc:2463] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:38.265513 991 leader_election.cc:304] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:38.265560 2179 consensus_queue.cc:1035] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:38.265786 2215 raft_consensus.cc:2798] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:38.265901 2215 raft_consensus.cc:695] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:38.266139 2215 consensus_queue.cc:237] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.267208 889 tablet_service.cc:1514] Processing DeleteTablet for tablet cbe30f9860e2492b90e8cedf8ff9daba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.267303 742 tablet_service.cc:1514] Processing DeleteTablet for tablet cbe30f9860e2492b90e8cedf8ff9daba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.267334 2317 leader_election.cc:290] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:38.267871 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet cbe30f9860e2492b90e8cedf8ff9daba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbe30f9860e2492b90e8cedf8ff9daba already in progress: creating tablet
I20250112 02:52:38.268028 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet cbe30f9860e2492b90e8cedf8ff9daba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.273953 989 leader_election.cc:304] T 4d25815a5df94e099e7481c48c3da793 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:38.274281 2326 raft_consensus.cc:2743] T 4d25815a5df94e099e7481c48c3da793 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:38.276146 854 leader_election.cc:343] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.276261 854 leader_election.cc:304] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:38.276481 2399 raft_consensus.cc:2743] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:38.276615 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_d561e1740cc545dc8aa4babb76b45c37 [id=de27aeff6b3a49b3956498ea5015ff33] alter complete (version 1)
I20250112 02:52:38.277007 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0dc78d722ac4687bfb3dce94d6db7f0" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:38.277209 919 raft_consensus.cc:2388] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f1e775296eb746db8b318f17d26820e6 in current term 1: Already voted for candidate 289afc116aa74408aa5bff74e11c50d8 in this term.
I20250112 02:52:38.277132 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0dc78d722ac4687bfb3dce94d6db7f0" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:38.277360 918 raft_consensus.cc:2388] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1e775296eb746db8b318f17d26820e6 in current term 1: Already voted for candidate 289afc116aa74408aa5bff74e11c50d8 in this term.
I20250112 02:52:38.277693 989 leader_election.cc:304] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:38.277978 2326 raft_consensus.cc:2743] T f0dc78d722ac4687bfb3dce94d6db7f0 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250112 02:52:38.278169 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2361bc6bd8164c2daa37be6b0845f4bd" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:38.284101 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c5079f87652a4ea58b366d1f9448fd4e consensus state has no leader
W20250112 02:52:38.287786 989 leader_election.cc:343] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: b8daf4ef3593410eae87672402292be6
W20250112 02:52:38.288142 989 leader_election.cc:343] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: b8daf4ef3593410eae87672402292be6
I20250112 02:52:38.288579 2399 consensus_queue.cc:1035] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:38.289152 854 leader_election.cc:304] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:38.289417 2399 raft_consensus.cc:2743] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:38.292946 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_69f363c4da234792886b544823b3214e [id=c7052afaf6284ef0972835c1f46cb829] alter complete (version 1)
I20250112 02:52:38.295542 2024 ts_tablet_manager.cc:1918] T 6cb76746b3a846c3aba531c04bf63966 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:38.295744 2024 log.cc:1198] T 6cb76746b3a846c3aba531c04bf63966 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/6cb76746b3a846c3aba531c04bf63966
I20250112 02:52:38.296116 2024 ts_tablet_manager.cc:1939] T 6cb76746b3a846c3aba531c04bf63966 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:38.296480 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 97) took 3760 ms. Trace:
I20250112 02:52:38.296742 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_4934f6022f2441eda88751236a5798be [id=5c4769bfcefd4fe0a0f3c2e315d50c32] alter complete (version 6)
I20250112 02:52:38.296623 2024 rpcz_store.cc:276] 0112 02:52:34.535670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.541228 (+ 5558us) service_pool.cc:225] Handling call
0112 02:52:38.204965 (+3663737us) ts_tablet_manager.cc:1145] Deleting tablet 6cb76746b3a846c3aba531c04bf63966
0112 02:52:38.204973 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.221789 (+ 16816us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.295398 (+ 73609us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.296466 (+ 1068us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12240,"tablet-delete.queue_time_us":3663619}
I20250112 02:52:38.297133 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 6cb76746b3a846c3aba531c04bf63966 (table default.Table_75db5f75b2874392b44dd580e0e52de9 [id=013bd2f66fe64ed593d8c0261ffa1c18]) successfully deleted
I20250112 02:52:38.297295 2024 tablet_replica.cc:331] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:38.297787 2024 raft_consensus.cc:2238] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:38.318710 2003 ts_tablet_manager.cc:1918] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:38.318838 2003 log.cc:1198] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/a12c5d58e6c246c983e8a533906ce77d
I20250112 02:52:38.319132 2003 ts_tablet_manager.cc:1939] T a12c5d58e6c246c983e8a533906ce77d P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:38.319432 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 84) took 4010 ms. Trace:
I20250112 02:52:38.319541 2003 rpcz_store.cc:276] 0112 02:52:34.308915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.308955 (+ 40us) service_pool.cc:225] Handling call
0112 02:52:38.211520 (+3902565us) ts_tablet_manager.cc:1145] Deleting tablet a12c5d58e6c246c983e8a533906ce77d
0112 02:52:38.211530 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.315757 (+104227us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.318574 (+ 2817us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.319403 (+ 829us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3902258}
I20250112 02:52:38.319937 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet a12c5d58e6c246c983e8a533906ce77d (table default.Table_f7bab317410d408aa363857a99be6e3c [id=d0be0fa53a79498e9160bfaa7d88837b]) successfully deleted
I20250112 02:52:38.327944 2399 raft_consensus.cc:491] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.328089 2399 raft_consensus.cc:513] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.328696 2399 leader_election.cc:290] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:38.329654 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec5346e13164274ae7bc3b6aa241f61" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:38.330152 855 leader_election.cc:343] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.339589 2003 tablet_replica.cc:331] T d4abf59d96414e5faa7cf75ba133832c P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:38.340274 1177 tablet_bootstrap.cc:492] T f05c8426ea3d4fe89fbe6c88c688019f P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:38.340396 1177 ts_tablet_manager.cc:1397] T f05c8426ea3d4fe89fbe6c88c688019f P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.147s user 0.002s sys 0.003s
I20250112 02:52:38.341055 1177 raft_consensus.cc:357] T f05c8426ea3d4fe89fbe6c88c688019f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.341225 1177 raft_consensus.cc:383] T f05c8426ea3d4fe89fbe6c88c688019f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.341284 1177 raft_consensus.cc:738] T f05c8426ea3d4fe89fbe6c88c688019f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.341485 1177 consensus_queue.cc:260] T f05c8426ea3d4fe89fbe6c88c688019f P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.301323 1173 tablet_bootstrap.cc:492] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:38.343742 1173 ts_tablet_manager.cc:1397] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.221s user 0.002s sys 0.003s
I20250112 02:52:38.344316 1173 raft_consensus.cc:357] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.344470 1173 raft_consensus.cc:383] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.344508 1173 raft_consensus.cc:738] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.344646 1173 consensus_queue.cc:260] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.345058 1173 ts_tablet_manager.cc:1428] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:38.345268 1173 tablet_bootstrap.cc:492] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:38.346421 1173 tablet_bootstrap.cc:654] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:38.302968 991 consensus_peers.cc:487] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d5b3df88a0c45dba879b3d506710614. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:38.347170 1177 ts_tablet_manager.cc:1428] T f05c8426ea3d4fe89fbe6c88c688019f P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
W20250112 02:52:38.347244 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 44298.7us
I20250112 02:52:38.347421 1177 tablet_bootstrap.cc:492] T e4d42f1611de49d6b6e5aaa9be907054 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:38.319180 1049 raft_consensus.cc:3054] T 4d25815a5df94e099e7481c48c3da793 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.348970 2003 raft_consensus.cc:2238] T d4abf59d96414e5faa7cf75ba133832c P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.349128 2003 raft_consensus.cc:2267] T d4abf59d96414e5faa7cf75ba133832c P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.329489 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec5346e13164274ae7bc3b6aa241f61" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:38.351095 1177 tablet_bootstrap.cc:654] T e4d42f1611de49d6b6e5aaa9be907054 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:38.351070 854 leader_election.cc:343] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.351176 854 leader_election.cc:304] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:38.351567 2399 raft_consensus.cc:2743] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:38.351954 2003 ts_tablet_manager.cc:1905] T d4abf59d96414e5faa7cf75ba133832c P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:38.352860 989 consensus_peers.cc:487] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ee98f055024429eafaa29bb961cab87. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:38.367285 2215 raft_consensus.cc:491] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.367506 2215 raft_consensus.cc:513] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.368046 2215 leader_election.cc:290] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.368503 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d42f1611de49d6b6e5aaa9be907054" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:38.368718 910 raft_consensus.cc:2463] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:38.369086 989 leader_election.cc:304] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:38.369379 2215 raft_consensus.cc:2798] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:38.369462 2215 raft_consensus.cc:491] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:38.369514 2215 raft_consensus.cc:3054] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.371793 2215 raft_consensus.cc:513] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.372838 1177 tablet_bootstrap.cc:492] T e4d42f1611de49d6b6e5aaa9be907054 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:38.372689 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d42f1611de49d6b6e5aaa9be907054" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:38.372792 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d42f1611de49d6b6e5aaa9be907054" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:38.373005 1177 ts_tablet_manager.cc:1397] T e4d42f1611de49d6b6e5aaa9be907054 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
W20250112 02:52:38.373438 991 leader_election.cc:343] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.373653 1177 raft_consensus.cc:357] T e4d42f1611de49d6b6e5aaa9be907054 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.373885 1177 raft_consensus.cc:383] T e4d42f1611de49d6b6e5aaa9be907054 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.373996 1177 raft_consensus.cc:738] T e4d42f1611de49d6b6e5aaa9be907054 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.373996 2215 leader_election.cc:290] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.374264 1177 consensus_queue.cc:260] T e4d42f1611de49d6b6e5aaa9be907054 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.374416 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d42f1611de49d6b6e5aaa9be907054" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:38.374583 910 raft_consensus.cc:3054] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.375747 1177 ts_tablet_manager.cc:1428] T e4d42f1611de49d6b6e5aaa9be907054 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 02:52:38.376086 1177 tablet_bootstrap.cc:492] T c4a3486ef011492c8a1df8d6278adedf P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:38.377120 910 raft_consensus.cc:2463] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:38.380328 1101 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:38.379930 775 raft_consensus.cc:2463] T e4d42f1611de49d6b6e5aaa9be907054 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:38.381721 1177 tablet_bootstrap.cc:654] T c4a3486ef011492c8a1df8d6278adedf P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.384318 989 leader_election.cc:304] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:38.385686 2215 raft_consensus.cc:2798] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:38.386281 2215 raft_consensus.cc:695] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:38.387053 2215 consensus_queue.cc:237] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.387936 2003 ts_tablet_manager.cc:1918] T d4abf59d96414e5faa7cf75ba133832c P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:38.388051 2003 log.cc:1198] T d4abf59d96414e5faa7cf75ba133832c P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/d4abf59d96414e5faa7cf75ba133832c
I20250112 02:52:38.390267 2399 raft_consensus.cc:491] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.390447 2399 raft_consensus.cc:513] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.391182 2399 leader_election.cc:290] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:38.391558 2003 ts_tablet_manager.cc:1939] T d4abf59d96414e5faa7cf75ba133832c P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:38.392174 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a32b424ec4431d88f6af45dd09089f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:38.392277 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 85) took 4081 ms. Trace:
I20250112 02:52:38.392365 2003 rpcz_store.cc:276] 0112 02:52:34.310367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.310406 (+ 39us) service_pool.cc:225] Handling call
0112 02:52:38.319817 (+4009411us) ts_tablet_manager.cc:1145] Deleting tablet d4abf59d96414e5faa7cf75ba133832c
0112 02:52:38.339525 (+ 19708us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.361121 (+ 21596us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.387816 (+ 26695us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.392224 (+ 4408us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4009148}
I20250112 02:52:38.392808 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet d4abf59d96414e5faa7cf75ba133832c (table default.Table_525750dad2b7461586d070dbaa74d0b1 [id=7d048aa907a74605964b68726ac5d759]) successfully deleted
W20250112 02:52:38.392952 854 leader_election.cc:343] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.393169 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a32b424ec4431d88f6af45dd09089f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:38.393658 855 leader_election.cc:343] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.393779 855 leader_election.cc:304] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:38.394145 2399 raft_consensus.cc:2743] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:38.393309 2003 tablet_replica.cc:331] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:38.395604 2024 raft_consensus.cc:2267] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.395812 2003 raft_consensus.cc:2238] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:38.396045 2003 pending_rounds.cc:62] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:38.396108 2003 pending_rounds.cc:69] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319864669765632 op_type: NO_OP noop_request { }
I20250112 02:52:38.396231 2003 raft_consensus.cc:2883] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:38.396323 2003 raft_consensus.cc:2267] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.396279 2024 ts_tablet_manager.cc:1905] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.401262 1177 tablet_bootstrap.cc:492] T c4a3486ef011492c8a1df8d6278adedf P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:38.401522 1177 ts_tablet_manager.cc:1397] T c4a3486ef011492c8a1df8d6278adedf P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20250112 02:52:38.402319 1177 raft_consensus.cc:357] T c4a3486ef011492c8a1df8d6278adedf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.402554 1177 raft_consensus.cc:383] T c4a3486ef011492c8a1df8d6278adedf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.402666 1177 raft_consensus.cc:738] T c4a3486ef011492c8a1df8d6278adedf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.402942 1177 consensus_queue.cc:260] T c4a3486ef011492c8a1df8d6278adedf P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.413048 1181 tablet_bootstrap.cc:492] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:38.413849 1177 ts_tablet_manager.cc:1428] T c4a3486ef011492c8a1df8d6278adedf P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250112 02:52:38.414062 1181 ts_tablet_manager.cc:1397] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.250s user 0.027s sys 0.052s
I20250112 02:52:38.414811 1177 tablet_bootstrap.cc:492] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:38.414813 1181 raft_consensus.cc:357] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.415005 1181 raft_consensus.cc:383] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.415073 1181 raft_consensus.cc:738] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.415266 1181 consensus_queue.cc:260] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.415757 1181 ts_tablet_manager.cc:1428] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:38.416047 1181 tablet_bootstrap.cc:492] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:38.419257 2003 ts_tablet_manager.cc:1905] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.429587 1173 tablet_bootstrap.cc:492] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:38.429733 1173 ts_tablet_manager.cc:1397] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.085s user 0.005s sys 0.005s
I20250112 02:52:38.430498 1173 raft_consensus.cc:357] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.430714 1173 raft_consensus.cc:383] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.430780 1173 raft_consensus.cc:738] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.431012 1173 consensus_queue.cc:260] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.431761 1173 ts_tablet_manager.cc:1428] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:38.421536 1177 tablet_bootstrap.cc:654] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.431980 1173 tablet_bootstrap.cc:492] T 7a34d03a80ae402db6607261a5c40b60 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:38.433805 1181 tablet_bootstrap.cc:654] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.443929 2496 raft_consensus.cc:491] T 6171b8ff2bfe41d598b0cb2535799a11 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.444052 2496 raft_consensus.cc:513] T 6171b8ff2bfe41d598b0cb2535799a11 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.444586 2496 leader_election.cc:290] T 6171b8ff2bfe41d598b0cb2535799a11 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:38.445223 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6171b8ff2bfe41d598b0cb2535799a11" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:38.445787 855 leader_election.cc:343] T 6171b8ff2bfe41d598b0cb2535799a11 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.446403 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6171b8ff2bfe41d598b0cb2535799a11" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:38.446977 854 leader_election.cc:343] T 6171b8ff2bfe41d598b0cb2535799a11 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.447073 854 leader_election.cc:304] T 6171b8ff2bfe41d598b0cb2535799a11 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:38.447515 2496 raft_consensus.cc:2743] T 6171b8ff2bfe41d598b0cb2535799a11 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:38.452401 2003 ts_tablet_manager.cc:1918] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:38.452515 2003 log.cc:1198] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/c8a7281c0c09442ebd0cc13b8496c86e
I20250112 02:52:38.452880 2003 ts_tablet_manager.cc:1939] T c8a7281c0c09442ebd0cc13b8496c86e P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:38.453622 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 86) took 4107 ms. Trace:
I20250112 02:52:38.453755 2003 rpcz_store.cc:276] 0112 02:52:34.345819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.345884 (+ 65us) service_pool.cc:225] Handling call
0112 02:52:38.393230 (+4047346us) ts_tablet_manager.cc:1145] Deleting tablet c8a7281c0c09442ebd0cc13b8496c86e
0112 02:52:38.393239 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.449181 (+ 55942us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.452325 (+ 3144us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.453603 (+ 1278us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4023416}
I20250112 02:52:38.454311 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet c8a7281c0c09442ebd0cc13b8496c86e (table default.Table_105e5d1a8ef846ffbcb1bdbe01bbae55 [id=082744a588e64fe8bafe0bc442876fc9]) successfully deleted
I20250112 02:52:38.456617 1181 tablet_bootstrap.cc:492] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:38.456779 1181 ts_tablet_manager.cc:1397] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.002s
I20250112 02:52:38.457338 2522 tablet.cc:1721] T 6117dc8b32a64535a2f4167a03bed0d2 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:38.460649 1181 raft_consensus.cc:357] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.460837 1181 raft_consensus.cc:383] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.460911 1181 raft_consensus.cc:738] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.461117 1181 consensus_queue.cc:260] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.461638 1181 ts_tablet_manager.cc:1428] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 02:52:38.463538 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:38.463621 2024 ts_tablet_manager.cc:1918] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:38.463748 2024 log.cc:1198] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/73136be25c79491784d3ea71aa9b27f2
I20250112 02:52:38.464166 2024 ts_tablet_manager.cc:1939] T 73136be25c79491784d3ea71aa9b27f2 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:38.465730 1181 tablet_bootstrap.cc:492] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:38.467227 1181 tablet_bootstrap.cc:654] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.468255 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 96) took 3932 ms. Trace:
I20250112 02:52:38.468348 2024 rpcz_store.cc:276] 0112 02:52:34.535599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.541211 (+ 5612us) service_pool.cc:225] Handling call
0112 02:52:38.297167 (+3755956us) ts_tablet_manager.cc:1145] Deleting tablet 73136be25c79491784d3ea71aa9b27f2
0112 02:52:38.297179 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.399199 (+102020us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.463460 (+ 64261us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.468236 (+ 4776us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3755763}
I20250112 02:52:38.468662 2024 tablet_replica.cc:331] T d45fce86e02f46d4a57af718fa2a62c8 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:38.468832 2024 raft_consensus.cc:2238] T d45fce86e02f46d4a57af718fa2a62c8 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.468947 2024 raft_consensus.cc:2267] T d45fce86e02f46d4a57af718fa2a62c8 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.469446 2024 ts_tablet_manager.cc:1905] T d45fce86e02f46d4a57af718fa2a62c8 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.475787 1177 tablet_bootstrap.cc:492] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:38.475937 1177 ts_tablet_manager.cc:1397] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.061s user 0.006s sys 0.000s
I20250112 02:52:38.476970 1173 tablet_bootstrap.cc:654] T 7a34d03a80ae402db6607261a5c40b60 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.479493 1177 raft_consensus.cc:357] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.479815 1177 raft_consensus.cc:383] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.479912 1177 raft_consensus.cc:738] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.480135 1177 consensus_queue.cc:260] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.484676 1173 tablet_bootstrap.cc:492] T 7a34d03a80ae402db6607261a5c40b60 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:38.484791 1173 ts_tablet_manager.cc:1397] T 7a34d03a80ae402db6607261a5c40b60 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.000s
I20250112 02:52:38.484969 1177 ts_tablet_manager.cc:1428] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250112 02:52:38.485581 1173 raft_consensus.cc:357] T 7a34d03a80ae402db6607261a5c40b60 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.485750 1173 raft_consensus.cc:383] T 7a34d03a80ae402db6607261a5c40b60 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.485805 1173 raft_consensus.cc:738] T 7a34d03a80ae402db6607261a5c40b60 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.485992 1173 consensus_queue.cc:260] T 7a34d03a80ae402db6607261a5c40b60 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.486553 1173 ts_tablet_manager.cc:1428] T 7a34d03a80ae402db6607261a5c40b60 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:38.486841 1173 tablet_bootstrap.cc:492] T 003a3b4705e943c6a900762ec11ddd47 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:38.487546 1177 tablet_bootstrap.cc:492] T b61f16f3d601436bb4b32344de28c1a7 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:38.489331 1173 tablet_bootstrap.cc:654] T 003a3b4705e943c6a900762ec11ddd47 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.468782 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 73136be25c79491784d3ea71aa9b27f2 (table default.Table_e3c5ba009c9f4301a3f99d51a8dcf687 [id=d5bf19dc7bff469ba06018568b796a63]) successfully deleted
I20250112 02:52:38.491794 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:38.493099 1177 tablet_bootstrap.cc:654] T b61f16f3d601436bb4b32344de28c1a7 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.493616 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:38.496158 2523 tablet.cc:1721] T 6117dc8b32a64535a2f4167a03bed0d2 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:38.496564 1181 tablet_bootstrap.cc:492] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:38.496716 1181 ts_tablet_manager.cc:1397] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20250112 02:52:38.496755 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:38.493865 2003 tablet_replica.cc:331] T 7a18132549374218893f35a4e4226b72 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:38.497057 2003 raft_consensus.cc:2238] T 7a18132549374218893f35a4e4226b72 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.497246 2003 raft_consensus.cc:2267] T 7a18132549374218893f35a4e4226b72 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.497781 2003 ts_tablet_manager.cc:1905] T 7a18132549374218893f35a4e4226b72 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.490279 2525 tablet.cc:1721] T 6117dc8b32a64535a2f4167a03bed0d2 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:38.498159 1181 raft_consensus.cc:357] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.498459 1181 raft_consensus.cc:383] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.498581 1181 raft_consensus.cc:738] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.498905 1181 consensus_queue.cc:260] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.499642 1181 ts_tablet_manager.cc:1428] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 02:52:38.500499 1181 tablet_bootstrap.cc:492] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:38.501935 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
I20250112 02:52:38.502174 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet f3771166b77e4f90aaa116ab1a9dc299 (table default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6]). Expected version 1 got 0
I20250112 02:52:38.502419 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:38.503154 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet fccff64af8ca4400afc116db98918119 (table default.Table_15bcb2c0f2a74458a1bed231fd3437fc [id=ca9c8281bf254f248076cb600e7c4f87]). Expected version 2 got 0
I20250112 02:52:38.503381 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet f924459d1150472aafb325a11825db23 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]). Expected version 3 got 0
I20250112 02:52:38.503719 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:38.506773 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:38.507050 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:38.507371 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:38.514283 1181 tablet_bootstrap.cc:654] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.522572 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_5d473cabff2e46e0b531fbce28d9d558 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 1 got 0
I20250112 02:52:38.522889 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:38.523223 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:38.525282 1177 tablet_bootstrap.cc:492] T b61f16f3d601436bb4b32344de28c1a7 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:38.525445 1177 ts_tablet_manager.cc:1397] T b61f16f3d601436bb4b32344de28c1a7 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20250112 02:52:38.526324 2522 tablet.cc:1721] T fccff64af8ca4400afc116db98918119 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:38.526360 1177 raft_consensus.cc:357] T b61f16f3d601436bb4b32344de28c1a7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.526585 1177 raft_consensus.cc:383] T b61f16f3d601436bb4b32344de28c1a7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.526679 1177 raft_consensus.cc:738] T b61f16f3d601436bb4b32344de28c1a7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.526958 1177 consensus_queue.cc:260] T b61f16f3d601436bb4b32344de28c1a7 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.529119 2024 ts_tablet_manager.cc:1918] T d45fce86e02f46d4a57af718fa2a62c8 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:38.529335 1177 ts_tablet_manager.cc:1428] T b61f16f3d601436bb4b32344de28c1a7 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250112 02:52:38.529346 2024 log.cc:1198] T d45fce86e02f46d4a57af718fa2a62c8 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/d45fce86e02f46d4a57af718fa2a62c8
I20250112 02:52:38.529493 2525 tablet.cc:1721] T fccff64af8ca4400afc116db98918119 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:38.529629 2024 ts_tablet_manager.cc:1939] T d45fce86e02f46d4a57af718fa2a62c8 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:38.534874 1181 tablet_bootstrap.cc:492] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:38.535104 1181 ts_tablet_manager.cc:1397] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.004s
I20250112 02:52:38.535172 2228 raft_consensus.cc:491] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.535269 2228 raft_consensus.cc:513] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.535840 2228 leader_election.cc:290] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:38.536576 991 leader_election.cc:343] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: a12c5d58e6c246c983e8a533906ce77d
I20250112 02:52:38.536808 1181 raft_consensus.cc:357] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:38.537067 989 leader_election.cc:343] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: a12c5d58e6c246c983e8a533906ce77d
I20250112 02:52:38.537158 989 leader_election.cc:304] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:38.537304 1181 raft_consensus.cc:383] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.537542 2215 raft_consensus.cc:2743] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:38.537515 1181 raft_consensus.cc:738] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.537673 2215 raft_consensus.cc:491] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 289afc116aa74408aa5bff74e11c50d8)
I20250112 02:52:38.537793 2215 raft_consensus.cc:513] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } }
I20250112 02:52:38.538336 2215 leader_election.cc:290] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
W20250112 02:52:38.539350 991 leader_election.cc:343] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: 1a8a178bafcb457f9b0e9587df560493
I20250112 02:52:38.538154 1181 consensus_queue.cc:260] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:38.531599 1177 tablet_bootstrap.cc:492] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:38.540596 1181 ts_tablet_manager.cc:1428] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 02:52:38.542225 1181 tablet_bootstrap.cc:492] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:38.542677 2228 raft_consensus.cc:491] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.542788 2228 raft_consensus.cc:513] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.543277 2228 leader_election.cc:290] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.543752 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05c8426ea3d4fe89fbe6c88c688019f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:38.543912 775 raft_consensus.cc:2463] T f05c8426ea3d4fe89fbe6c88c688019f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:38.544617 991 leader_election.cc:304] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
W20250112 02:52:38.544900 989 leader_election.cc:343] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 1a8a178bafcb457f9b0e9587df560493
I20250112 02:52:38.544974 989 leader_election.cc:304] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:38.545126 2215 raft_consensus.cc:2798] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:38.545192 2215 raft_consensus.cc:491] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:38.545255 2215 raft_consensus.cc:3054] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.545305 2228 raft_consensus.cc:2743] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250112 02:52:38.547963 989 leader_election.cc:343] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: f05c8426ea3d4fe89fbe6c88c688019f
I20250112 02:52:38.547891 2215 raft_consensus.cc:513] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.548790 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05c8426ea3d4fe89fbe6c88c688019f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:38.548950 775 raft_consensus.cc:3054] T f05c8426ea3d4fe89fbe6c88c688019f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.550063 1173 tablet_bootstrap.cc:492] T 003a3b4705e943c6a900762ec11ddd47 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:38.550136 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 98) took 4013 ms. Trace:
I20250112 02:52:38.550184 1173 ts_tablet_manager.cc:1397] T 003a3b4705e943c6a900762ec11ddd47 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.005s
I20250112 02:52:38.550215 2024 rpcz_store.cc:276] 0112 02:52:34.536776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.541354 (+ 4578us) service_pool.cc:225] Handling call
0112 02:52:38.468619 (+3927265us) ts_tablet_manager.cc:1145] Deleting tablet d45fce86e02f46d4a57af718fa2a62c8
0112 02:52:38.468628 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.505273 (+ 36645us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.529043 (+ 23770us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.550120 (+ 21077us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3927122}
I20250112 02:52:38.550597 2024 tablet_replica.cc:331] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:38.550755 2024 raft_consensus.cc:2238] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.550897 2024 raft_consensus.cc:2267] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.550766 1173 raft_consensus.cc:357] T 003a3b4705e943c6a900762ec11ddd47 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.551121 1173 raft_consensus.cc:383] T 003a3b4705e943c6a900762ec11ddd47 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.551168 1173 raft_consensus.cc:738] T 003a3b4705e943c6a900762ec11ddd47 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.551394 2024 ts_tablet_manager.cc:1905] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.551344 1173 consensus_queue.cc:260] T 003a3b4705e943c6a900762ec11ddd47 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.552924 2215 leader_election.cc:290] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.551398 775 raft_consensus.cc:2463] T f05c8426ea3d4fe89fbe6c88c688019f P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
W20250112 02:52:38.553517 989 leader_election.cc:343] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: f05c8426ea3d4fe89fbe6c88c688019f
I20250112 02:52:38.553745 991 leader_election.cc:304] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:38.554112 2215 raft_consensus.cc:2798] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:38.554316 2215 raft_consensus.cc:695] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:38.554569 2215 consensus_queue.cc:237] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.556941 1181 tablet_bootstrap.cc:654] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.551764 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet d45fce86e02f46d4a57af718fa2a62c8 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]) successfully deleted
W20250112 02:52:38.563097 989 consensus_peers.cc:487] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: daef3a58605e40978f7389698b22265d. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:38.563475 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_5e7392cb5a68412f97bddb2677986918 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 1 got 0
I20250112 02:52:38.565244 1177 tablet_bootstrap.cc:654] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.565732 1173 ts_tablet_manager.cc:1428] T 003a3b4705e943c6a900762ec11ddd47 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20250112 02:52:38.565953 1173 tablet_bootstrap.cc:492] T b61f16f3d601436bb4b32344de28c1a7 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:38.568027 2496 raft_consensus.cc:491] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.568137 2496 raft_consensus.cc:513] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.568434 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 1 got 0
I20250112 02:52:38.568660 2496 leader_election.cc:290] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:38.568933 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:38.569090 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 6117dc8b32a64535a2f4167a03bed0d2 (table default.Table_365739770a11434dbf260ea912e7ea81 [id=868adb627654450c936ef198e49d853b]). Expected version 1 got 0
I20250112 02:52:38.569566 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79eb61cedd544c49869471eebe2fd237" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:38.570097 854 leader_election.cc:343] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.574635 2024 ts_tablet_manager.cc:1918] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:38.574741 2024 log.cc:1198] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/2730634a5cfd4c2fb5c4e30100c556d4
I20250112 02:52:38.575074 2024 ts_tablet_manager.cc:1939] T 2730634a5cfd4c2fb5c4e30100c556d4 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:38.589150 2496 raft_consensus.cc:491] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.589286 2496 raft_consensus.cc:513] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:38.589872 2496 leader_election.cc:290] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:38.590680 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2052b4536f9745f0a785f330d706c1cf" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:38.590862 1181 tablet_bootstrap.cc:492] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:38.590977 1181 ts_tablet_manager.cc:1397] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
W20250112 02:52:38.591197 854 leader_election.cc:343] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.591722 1181 raft_consensus.cc:357] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.591902 1181 raft_consensus.cc:383] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.591954 1181 raft_consensus.cc:738] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.592149 1181 consensus_queue.cc:260] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.565392 2523 tablet.cc:1721] T fccff64af8ca4400afc116db98918119 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:38.570847 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79eb61cedd544c49869471eebe2fd237" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:38.593544 855 leader_election.cc:343] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.593629 855 leader_election.cc:304] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:38.581123 2003 ts_tablet_manager.cc:1918] T 7a18132549374218893f35a4e4226b72 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250112 02:52:38.588244 714 consensus_peers.cc:487] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2730634a5cfd4c2fb5c4e30100c556d4. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:38.591526 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2052b4536f9745f0a785f330d706c1cf" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:38.594878 2496 raft_consensus.cc:2743] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:38.595161 855 leader_election.cc:343] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.595240 855 leader_election.cc:304] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:38.595306 1181 ts_tablet_manager.cc:1428] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:38.595405 2496 raft_consensus.cc:2743] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:38.595562 2003 log.cc:1198] T 7a18132549374218893f35a4e4226b72 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/7a18132549374218893f35a4e4226b72
I20250112 02:52:38.595721 1181 tablet_bootstrap.cc:492] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:38.595952 2003 ts_tablet_manager.cc:1939] T 7a18132549374218893f35a4e4226b72 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:38.596625 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 88) took 4233 ms. Trace:
I20250112 02:52:38.596729 2003 rpcz_store.cc:276] 0112 02:52:34.362733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.362943 (+ 210us) service_pool.cc:225] Handling call
0112 02:52:38.493823 (+4130880us) ts_tablet_manager.cc:1145] Deleting tablet 7a18132549374218893f35a4e4226b72
0112 02:52:38.493832 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.500784 (+ 6952us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.581035 (+ 80251us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.596604 (+ 15569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4118306}
I20250112 02:52:38.597100 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 7a18132549374218893f35a4e4226b72 (table default.Table_3fcca85a33054d7ab388f29bba85fe35 [id=18415100e94043e99f9884b853158ae1]) successfully deleted
I20250112 02:52:38.597463 2003 tablet_replica.cc:331] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:38.597499 1181 tablet_bootstrap.cc:654] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.597741 2003 raft_consensus.cc:2238] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:38.597977 2003 pending_rounds.cc:62] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:38.598074 2003 pending_rounds.cc:69] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319866166120448 op_type: NO_OP noop_request { }
I20250112 02:52:38.598218 2003 raft_consensus.cc:2883] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:38.598335 2003 raft_consensus.cc:2267] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.600406 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:38.601037 454 catalog_manager.cc:5526] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:38.601846 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:38.602010 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:38.602523 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:38.602663 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 99) took 4065 ms. Trace:
I20250112 02:52:38.602872 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:38.602807 2024 rpcz_store.cc:276] 0112 02:52:34.536901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.541417 (+ 4516us) service_pool.cc:225] Handling call
0112 02:52:38.550555 (+4009138us) ts_tablet_manager.cc:1145] Deleting tablet 2730634a5cfd4c2fb5c4e30100c556d4
0112 02:52:38.550563 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.564424 (+ 13861us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.574550 (+ 10126us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.602063 (+ 27513us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4008992}
I20250112 02:52:38.603029 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
I20250112 02:52:38.603361 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 2730634a5cfd4c2fb5c4e30100c556d4 (table default.Table_a98f3e8602ec4ba4a771b14df9c07131 [id=274b4ed84aa7490dbb78270b48150b73]) successfully deleted
I20250112 02:52:38.603201 454 catalog_manager.cc:5526] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:38.603628 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:38.603971 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5cc414d6420c41839621ab858067548b (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]). Expected version 1 got 0
I20250112 02:52:38.604135 2024 tablet_replica.cc:331] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:38.604179 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:38.604403 2024 raft_consensus.cc:2238] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.604575 2024 raft_consensus.cc:2267] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.604605 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:38.604861 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:38.605878 2024 ts_tablet_manager.cc:1905] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.613372 1173 tablet_bootstrap.cc:654] T b61f16f3d601436bb4b32344de28c1a7 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.621057 1181 tablet_bootstrap.cc:492] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:38.621176 1181 ts_tablet_manager.cc:1397] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.002s
I20250112 02:52:38.621654 2003 ts_tablet_manager.cc:1905] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.621932 1181 raft_consensus.cc:357] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.622156 1181 raft_consensus.cc:383] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.622216 1181 raft_consensus.cc:738] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.631115 1177 tablet_bootstrap.cc:492] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:38.631227 1177 ts_tablet_manager.cc:1397] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.100s user 0.004s sys 0.000s
I20250112 02:52:38.632422 1177 raft_consensus.cc:357] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.632633 1177 raft_consensus.cc:383] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.632704 1177 raft_consensus.cc:738] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.632954 2496 raft_consensus.cc:491] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.632887 1177 consensus_queue.cc:260] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.633050 2496 raft_consensus.cc:513] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.633330 1177 ts_tablet_manager.cc:1428] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 02:52:38.633569 1177 tablet_bootstrap.cc:492] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:38.633549 2496 leader_election.cc:290] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:38.634020 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e477fc157aed4563923d85d66014293a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:38.634383 854 leader_election.cc:343] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.634594 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e477fc157aed4563923d85d66014293a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:38.605471 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:38.634802 1177 tablet_bootstrap.cc:654] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:38.635116 855 leader_election.cc:343] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.635205 855 leader_election.cc:304] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:38.635689 2496 raft_consensus.cc:2743] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:38.635908 1173 tablet_bootstrap.cc:492] T b61f16f3d601436bb4b32344de28c1a7 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:38.636018 1173 ts_tablet_manager.cc:1397] T b61f16f3d601436bb4b32344de28c1a7 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.070s user 0.005s sys 0.008s
I20250112 02:52:38.636353 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet f31b777e7e0f43a28839d8c48c6a6590 (table default.Table_5fb592e41cc4406aa27546e7b41fdb88 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 1 got 0
I20250112 02:52:38.636581 1173 raft_consensus.cc:357] T b61f16f3d601436bb4b32344de28c1a7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.636746 1173 raft_consensus.cc:383] T b61f16f3d601436bb4b32344de28c1a7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.636796 1173 raft_consensus.cc:738] T b61f16f3d601436bb4b32344de28c1a7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.636991 1173 consensus_queue.cc:260] T b61f16f3d601436bb4b32344de28c1a7 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.637252 1181 consensus_queue.cc:260] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.637598 1173 ts_tablet_manager.cc:1428] T b61f16f3d601436bb4b32344de28c1a7 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 02:52:38.637744 1181 ts_tablet_manager.cc:1428] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250112 02:52:38.637840 1173 tablet_bootstrap.cc:492] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:38.637962 1181 tablet_bootstrap.cc:492] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:38.638527 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:38.639195 1173 tablet_bootstrap.cc:654] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.639256 1181 tablet_bootstrap.cc:654] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.641115 454 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:38.644068 1173 tablet_bootstrap.cc:492] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:38.644176 1173 ts_tablet_manager.cc:1397] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20250112 02:52:38.644779 1173 raft_consensus.cc:357] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.644928 1173 raft_consensus.cc:383] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.644976 1173 raft_consensus.cc:738] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.645159 1173 consensus_queue.cc:260] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.645640 1173 ts_tablet_manager.cc:1428] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250112 02:52:38.645876 1173 tablet_bootstrap.cc:492] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:38.647418 1173 tablet_bootstrap.cc:654] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.649047 2496 raft_consensus.cc:491] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.649147 2496 raft_consensus.cc:513] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.649617 2496 leader_election.cc:290] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:38.650019 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6650515ea540a8bbf21e2f88363360" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:38.650373 854 leader_election.cc:343] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.650895 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6650515ea540a8bbf21e2f88363360" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:38.651397 855 leader_election.cc:343] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.651576 855 leader_election.cc:304] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:38.651961 2496 raft_consensus.cc:2743] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:38.653775 1181 tablet_bootstrap.cc:492] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:38.653896 1181 ts_tablet_manager.cc:1397] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20250112 02:52:38.654260 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_00028054ce7c431b96d17d8f6f8856a7 [id=f8cd4289f89a4a748c26a7a01236406e] alter complete (version 3)
I20250112 02:52:38.656479 2024 ts_tablet_manager.cc:1918] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:38.656555 2024 log.cc:1198] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/a12c5d58e6c246c983e8a533906ce77d
I20250112 02:52:38.656805 2024 ts_tablet_manager.cc:1939] T a12c5d58e6c246c983e8a533906ce77d P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:38.658603 2215 consensus_queue.cc:1035] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:38.659516 1181 raft_consensus.cc:357] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.660303 1181 raft_consensus.cc:383] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.660365 1181 raft_consensus.cc:738] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.660569 1181 consensus_queue.cc:260] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.661612 1181 ts_tablet_manager.cc:1428] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250112 02:52:38.661849 1181 tablet_bootstrap.cc:492] T 7e1954e7859f422f9690fe4a9384a8bb P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:38.663321 1181 tablet_bootstrap.cc:654] T 7e1954e7859f422f9690fe4a9384a8bb P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.664233 1177 tablet_bootstrap.cc:492] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:38.664357 1177 ts_tablet_manager.cc:1397] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.003s
I20250112 02:52:38.665124 1177 raft_consensus.cc:357] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.665321 1177 raft_consensus.cc:383] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.665372 1177 raft_consensus.cc:738] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.665634 1177 consensus_queue.cc:260] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.666272 1177 ts_tablet_manager.cc:1428] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:38.666519 1177 tablet_bootstrap.cc:492] T b9c006281dc949119951c74e23167d7f P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:38.668175 1177 tablet_bootstrap.cc:654] T b9c006281dc949119951c74e23167d7f P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.675441 2317 consensus_queue.cc:1035] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:38.678056 2522 tablet.cc:1721] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:38.678148 2496 raft_consensus.cc:491] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.678280 2496 raft_consensus.cc:513] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.678507 2523 tablet.cc:1721] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:38.678860 2496 leader_election.cc:290] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:38.679100 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea18666ec324403b9f10a099688438b8" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:38.679338 1173 tablet_bootstrap.cc:492] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:38.679316 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea18666ec324403b9f10a099688438b8" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:38.679438 1173 ts_tablet_manager.cc:1397] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.001s
W20250112 02:52:38.679764 854 leader_election.cc:343] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:38.679931 855 leader_election.cc:343] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.680012 855 leader_election.cc:304] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:38.679986 1173 raft_consensus.cc:357] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.680284 2496 raft_consensus.cc:2743] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:38.681828 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_96d1a09a902340ff8e05efb75faf3854 [id=ad0be37e6b6f4b9597dcb4cbd50d5b12] alter complete (version 3)
I20250112 02:52:38.680119 1173 raft_consensus.cc:383] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.685267 1173 raft_consensus.cc:738] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.685614 1173 consensus_queue.cc:260] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.686293 1173 ts_tablet_manager.cc:1428] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250112 02:52:38.686609 1173 tablet_bootstrap.cc:492] T 763f77385e404c149c16edabe469c299 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:38.689862 1181 tablet_bootstrap.cc:492] T 7e1954e7859f422f9690fe4a9384a8bb P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:38.689983 1181 ts_tablet_manager.cc:1397] T 7e1954e7859f422f9690fe4a9384a8bb P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.002s
I20250112 02:52:38.691123 1181 raft_consensus.cc:357] T 7e1954e7859f422f9690fe4a9384a8bb P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.691267 1181 raft_consensus.cc:383] T 7e1954e7859f422f9690fe4a9384a8bb P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.691309 1181 raft_consensus.cc:738] T 7e1954e7859f422f9690fe4a9384a8bb P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.693859 2525 tablet.cc:1721] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:38.693848 1181 consensus_queue.cc:260] T 7e1954e7859f422f9690fe4a9384a8bb P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.695030 1181 ts_tablet_manager.cc:1428] T 7e1954e7859f422f9690fe4a9384a8bb P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 02:52:38.695001 2451 consensus_queue.cc:1035] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:38.695230 1181 tablet_bootstrap.cc:492] T bd5b3c712c0f448ea9b14ab93dd731ab P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:38.697780 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_009a6976f6db4f218e3c3373368a58d7 [id=c5005669bb5f4c029d673bd39cc1535d] alter complete (version 1)
I20250112 02:52:38.698575 1177 tablet_bootstrap.cc:492] T b9c006281dc949119951c74e23167d7f P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:38.698712 1177 ts_tablet_manager.cc:1397] T b9c006281dc949119951c74e23167d7f P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.003s
I20250112 02:52:38.699262 1177 raft_consensus.cc:357] T b9c006281dc949119951c74e23167d7f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.699501 1177 raft_consensus.cc:383] T b9c006281dc949119951c74e23167d7f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.699600 1177 raft_consensus.cc:738] T b9c006281dc949119951c74e23167d7f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.699834 1177 consensus_queue.cc:260] T b9c006281dc949119951c74e23167d7f P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.700345 1177 ts_tablet_manager.cc:1428] T b9c006281dc949119951c74e23167d7f P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:38.700587 1177 tablet_bootstrap.cc:492] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:38.702090 1177 tablet_bootstrap.cc:654] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.702219 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 100) took 4165 ms. Trace:
I20250112 02:52:38.702713 2024 rpcz_store.cc:276] 0112 02:52:34.536997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.541508 (+ 4511us) service_pool.cc:225] Handling call
0112 02:52:38.603683 (+4062175us) ts_tablet_manager.cc:1145] Deleting tablet a12c5d58e6c246c983e8a533906ce77d
0112 02:52:38.603693 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.608790 (+ 5097us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.656392 (+ 47602us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.702200 (+ 45808us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4062041}
I20250112 02:52:38.703199 2024 tablet_replica.cc:331] T b61f16f3d601436bb4b32344de28c1a7 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:38.703392 2024 raft_consensus.cc:2238] T b61f16f3d601436bb4b32344de28c1a7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.704146 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet a12c5d58e6c246c983e8a533906ce77d (table default.Table_f7bab317410d408aa363857a99be6e3c [id=d0be0fa53a79498e9160bfaa7d88837b]) successfully deleted
I20250112 02:52:38.706290 2496 raft_consensus.cc:491] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.706396 2496 raft_consensus.cc:513] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.707146 2496 leader_election.cc:290] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:38.707669 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_83177e7dc14645dbb0e5ef573e80e72f [id=2c077da077834ae4bcc84efdd9e89120] alter complete (version 1)
I20250112 02:52:38.708386 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61eac6b572ff47c185768bee7490edf7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:38.708879 854 leader_election.cc:343] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.709887 1173 tablet_bootstrap.cc:654] T 763f77385e404c149c16edabe469c299 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.711891 1181 tablet_bootstrap.cc:654] T bd5b3c712c0f448ea9b14ab93dd731ab P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.702628 2003 ts_tablet_manager.cc:1918] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:38.713959 2003 log.cc:1198] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/d45fce86e02f46d4a57af718fa2a62c8
I20250112 02:52:38.714270 2003 ts_tablet_manager.cc:1939] T d45fce86e02f46d4a57af718fa2a62c8 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:38.707881 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61eac6b572ff47c185768bee7490edf7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:38.714699 855 leader_election.cc:343] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.714795 855 leader_election.cc:304] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:38.714928 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 87) took 4352 ms. Trace:
I20250112 02:52:38.714988 2003 rpcz_store.cc:276] 0112 02:52:34.362633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.362677 (+ 44us) service_pool.cc:225] Handling call
0112 02:52:38.597402 (+4234725us) ts_tablet_manager.cc:1145] Deleting tablet d45fce86e02f46d4a57af718fa2a62c8
0112 02:52:38.597411 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.659737 (+ 62326us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.702549 (+ 42812us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.714913 (+ 12364us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4152154}
I20250112 02:52:38.715371 2003 tablet_replica.cc:331] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:38.715576 2003 raft_consensus.cc:2238] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:38.715584 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet d45fce86e02f46d4a57af718fa2a62c8 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]) successfully deleted
I20250112 02:52:38.715866 2003 raft_consensus.cc:2267] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.716001 2496 raft_consensus.cc:2743] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:38.716243 2024 raft_consensus.cc:2267] T b61f16f3d601436bb4b32344de28c1a7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.716352 2003 ts_tablet_manager.cc:1905] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.720890 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_1216acc1fa6b4e51a55a89ffe22bcc9a [id=b5457e4343ee43618c8309e737b69ec3] alter complete (version 1)
I20250112 02:52:38.721432 2024 ts_tablet_manager.cc:1905] T b61f16f3d601436bb4b32344de28c1a7 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.723968 1177 tablet_bootstrap.cc:492] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:38.724112 1177 ts_tablet_manager.cc:1397] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.001s
I20250112 02:52:38.724814 1177 raft_consensus.cc:357] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.725044 1177 raft_consensus.cc:383] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.725138 1177 raft_consensus.cc:738] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.725574 1177 consensus_queue.cc:260] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.726137 1177 ts_tablet_manager.cc:1428] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:38.727604 1177 tablet_bootstrap.cc:492] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:38.729080 1181 tablet_bootstrap.cc:492] T bd5b3c712c0f448ea9b14ab93dd731ab P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:38.729229 1181 ts_tablet_manager.cc:1397] T bd5b3c712c0f448ea9b14ab93dd731ab P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20250112 02:52:38.729862 1181 raft_consensus.cc:357] T bd5b3c712c0f448ea9b14ab93dd731ab P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.730083 1181 raft_consensus.cc:383] T bd5b3c712c0f448ea9b14ab93dd731ab P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.730121 2024 ts_tablet_manager.cc:1918] T b61f16f3d601436bb4b32344de28c1a7 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:38.730172 1181 raft_consensus.cc:738] T bd5b3c712c0f448ea9b14ab93dd731ab P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.730360 1181 consensus_queue.cc:260] T bd5b3c712c0f448ea9b14ab93dd731ab P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.734100 1181 ts_tablet_manager.cc:1428] T bd5b3c712c0f448ea9b14ab93dd731ab P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250112 02:52:38.731088 1173 tablet_bootstrap.cc:492] T 763f77385e404c149c16edabe469c299 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:38.734320 1173 ts_tablet_manager.cc:1397] T 763f77385e404c149c16edabe469c299 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20250112 02:52:38.734437 1181 tablet_bootstrap.cc:492] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:38.734992 1173 raft_consensus.cc:357] T 763f77385e404c149c16edabe469c299 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.735211 1173 raft_consensus.cc:383] T 763f77385e404c149c16edabe469c299 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.735296 1173 raft_consensus.cc:738] T 763f77385e404c149c16edabe469c299 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.735580 1173 consensus_queue.cc:260] T 763f77385e404c149c16edabe469c299 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.735832 917 raft_consensus.cc:3054] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.736263 1173 ts_tablet_manager.cc:1428] T 763f77385e404c149c16edabe469c299 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:38.736485 1173 tablet_bootstrap.cc:492] T 7320ac55dbfa47fe851fff94d32e1ffe P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:38.736608 2003 ts_tablet_manager.cc:1918] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:38.736681 2003 log.cc:1198] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/2730634a5cfd4c2fb5c4e30100c556d4
I20250112 02:52:38.736953 2003 ts_tablet_manager.cc:1939] T 2730634a5cfd4c2fb5c4e30100c556d4 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:38.737517 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 89) took 4311 ms. Trace:
I20250112 02:52:38.737578 2003 rpcz_store.cc:276] 0112 02:52:34.426555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.426603 (+ 48us) service_pool.cc:225] Handling call
0112 02:52:38.715331 (+4288728us) ts_tablet_manager.cc:1145] Deleting tablet 2730634a5cfd4c2fb5c4e30100c556d4
0112 02:52:38.715341 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.719045 (+ 3704us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.736542 (+ 17497us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.737501 (+ 959us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4267252}
I20250112 02:52:38.737969 1173 tablet_bootstrap.cc:654] T 7320ac55dbfa47fe851fff94d32e1ffe P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.738466 1177 tablet_bootstrap.cc:654] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.738579 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 2730634a5cfd4c2fb5c4e30100c556d4 (table default.Table_a98f3e8602ec4ba4a771b14df9c07131 [id=274b4ed84aa7490dbb78270b48150b73]) successfully deleted
I20250112 02:52:38.739087 2451 consensus_queue.cc:1035] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:38.739230 2024 log.cc:1198] T b61f16f3d601436bb4b32344de28c1a7 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/b61f16f3d601436bb4b32344de28c1a7
I20250112 02:52:38.739413 1181 tablet_bootstrap.cc:654] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.739557 2024 ts_tablet_manager.cc:1939] T b61f16f3d601436bb4b32344de28c1a7 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:38.741763 2003 tablet_replica.cc:331] T b8daf4ef3593410eae87672402292be6 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:38.741931 2003 raft_consensus.cc:2238] T b8daf4ef3593410eae87672402292be6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.742040 2003 raft_consensus.cc:2267] T b8daf4ef3593410eae87672402292be6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.742432 2003 ts_tablet_manager.cc:1905] T b8daf4ef3593410eae87672402292be6 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.742924 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 101) took 4205 ms. Trace:
I20250112 02:52:38.742992 2024 rpcz_store.cc:276] 0112 02:52:34.537107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.541585 (+ 4478us) service_pool.cc:225] Handling call
0112 02:52:38.703161 (+4161576us) ts_tablet_manager.cc:1145] Deleting tablet b61f16f3d601436bb4b32344de28c1a7
0112 02:52:38.703169 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.726940 (+ 23771us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.730044 (+ 3104us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.742907 (+ 12863us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4161385}
I20250112 02:52:38.743366 2024 tablet_replica.cc:331] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:38.743420 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet b61f16f3d601436bb4b32344de28c1a7 (table default.Table_dafd4ddc8ab34adfa999f9340216f765 [id=da4d2d7725904d5bb4a6bde668df00ef]) successfully deleted
I20250112 02:52:38.743580 2024 raft_consensus.cc:2238] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.743713 2024 raft_consensus.cc:2267] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.744297 2024 ts_tablet_manager.cc:1905] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.754683 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_7cbe570a89784c18b4a14a8242062071 [id=fee2f9e65e314ce9a1fc1fb6dfcde317] alter complete (version 1)
I20250112 02:52:38.755702 1181 tablet_bootstrap.cc:492] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:38.755717 1177 tablet_bootstrap.cc:492] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:38.755810 1181 ts_tablet_manager.cc:1397] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20250112 02:52:38.755949 1177 ts_tablet_manager.cc:1397] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20250112 02:52:38.756397 1181 raft_consensus.cc:357] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.756587 1181 raft_consensus.cc:383] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.756649 1181 raft_consensus.cc:738] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.756862 1181 consensus_queue.cc:260] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.756901 1177 raft_consensus.cc:357] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.757192 1177 raft_consensus.cc:383] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.757284 1177 raft_consensus.cc:738] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.757301 1181 ts_tablet_manager.cc:1428] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 02:52:38.757501 1177 consensus_queue.cc:260] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.757598 1181 tablet_bootstrap.cc:492] T 9ded8bcdda914ea6bc4f9bb92d413dee P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:38.757999 1177 ts_tablet_manager.cc:1428] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:38.758280 1177 tablet_bootstrap.cc:492] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:38.759001 1181 tablet_bootstrap.cc:654] T 9ded8bcdda914ea6bc4f9bb92d413dee P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.759610 1173 tablet_bootstrap.cc:492] T 7320ac55dbfa47fe851fff94d32e1ffe P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:38.759773 1173 ts_tablet_manager.cc:1397] T 7320ac55dbfa47fe851fff94d32e1ffe P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.001s
I20250112 02:52:38.760226 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_57b3861e9f704dc09d466241c1926fc1 [id=9807191b6e5c4d478cd47a54464fb9ce] alter complete (version 2)
I20250112 02:52:38.760401 1173 raft_consensus.cc:357] T 7320ac55dbfa47fe851fff94d32e1ffe P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.760560 1173 raft_consensus.cc:383] T 7320ac55dbfa47fe851fff94d32e1ffe P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.760612 1173 raft_consensus.cc:738] T 7320ac55dbfa47fe851fff94d32e1ffe P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.760824 1173 consensus_queue.cc:260] T 7320ac55dbfa47fe851fff94d32e1ffe P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.761456 1173 ts_tablet_manager.cc:1428] T 7320ac55dbfa47fe851fff94d32e1ffe P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 02:52:38.761739 1173 tablet_bootstrap.cc:492] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:38.763203 1173 tablet_bootstrap.cc:654] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.764600 1177 tablet_bootstrap.cc:654] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.764757 2003 ts_tablet_manager.cc:1918] T b8daf4ef3593410eae87672402292be6 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:38.764843 2003 log.cc:1198] T b8daf4ef3593410eae87672402292be6 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/b8daf4ef3593410eae87672402292be6
I20250112 02:52:38.765059 2003 ts_tablet_manager.cc:1939] T b8daf4ef3593410eae87672402292be6 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:38.766435 742 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d411f906b4d4eb68343bdcc493e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.766538 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1954e7859f422f9690fe4a9384a8bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.767829 742 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5b3c712c0f448ea9b14ab93dd731ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.768039 742 tablet_service.cc:1514] Processing DeleteTablet for tablet bb217afc08304c0ea35dee3120347ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.768210 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0c2fad4f3243238641aae56b305bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.768328 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d0f67141dc46b6a87ca0ad080d5bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.768455 750 tablet_service.cc:1514] Processing DeleteTablet for tablet d659c5a4f7a446c6aaf2283f65b45106 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.768595 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 416fb6820cd54e118950c68d381f51b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.768718 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 161fe26b9c0e4543b7aeb0021506d72a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.768822 748 tablet_service.cc:1514] Processing DeleteTablet for tablet f7cd0927c6a4401cbe69f55d4db9963a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.768918 747 tablet_service.cc:1514] Processing DeleteTablet for tablet aadb2ab1d2024f889841ac91e065e064 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.769027 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c639a2bcd4740e4be7b2b162004669a with delete_type TABLET_DATA_DELETED (replaced by tablet 5d0c3191973a477e949fbf9dfcca963f) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.769150 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 87759e8b99b04e628badc8f60cb38bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.769264 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 70715db0394d46d9b6b289160b00be7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.769696 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 54215c55966a46e1ab96e352c4e3764e with delete_type TABLET_DATA_DELETED (replaced by tablet a470f84d50df4cf0b4a1e5d248b4c32a) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.770000 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f22a7b1b804661af3b7e785a833739 with delete_type TABLET_DATA_DELETED (replaced by tablet 3d662e0ce13f447595e5f92b94de3018) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.773149 1181 tablet_bootstrap.cc:492] T 9ded8bcdda914ea6bc4f9bb92d413dee P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:38.773317 1181 ts_tablet_manager.cc:1397] T 9ded8bcdda914ea6bc4f9bb92d413dee P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20250112 02:52:38.773787 2024 ts_tablet_manager.cc:1918] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:38.773890 2024 log.cc:1198] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/1b184806b4884cd3b0a002905a1e57af
I20250112 02:52:38.774060 1181 raft_consensus.cc:357] T 9ded8bcdda914ea6bc4f9bb92d413dee P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.774168 2024 ts_tablet_manager.cc:1939] T 1b184806b4884cd3b0a002905a1e57af P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:38.774271 1181 raft_consensus.cc:383] T 9ded8bcdda914ea6bc4f9bb92d413dee P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.774451 1181 raft_consensus.cc:738] T 9ded8bcdda914ea6bc4f9bb92d413dee P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.774680 1181 consensus_queue.cc:260] T 9ded8bcdda914ea6bc4f9bb92d413dee P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.774820 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 91) took 4239 ms. Trace:
I20250112 02:52:38.774906 2024 rpcz_store.cc:276] 0112 02:52:34.535112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.539975 (+ 4863us) service_pool.cc:225] Handling call
0112 02:52:38.743327 (+4203352us) ts_tablet_manager.cc:1145] Deleting tablet 1b184806b4884cd3b0a002905a1e57af
0112 02:52:38.743335 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.750122 (+ 6787us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.773703 (+ 23581us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.774803 (+ 1100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4201433}
I20250112 02:52:38.775192 2024 tablet_replica.cc:331] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:38.775282 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 1b184806b4884cd3b0a002905a1e57af (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]) successfully deleted
I20250112 02:52:38.775360 2024 raft_consensus.cc:2238] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.775494 2024 raft_consensus.cc:2267] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.775569 1181 ts_tablet_manager.cc:1428] T 9ded8bcdda914ea6bc4f9bb92d413dee P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:38.775794 1181 tablet_bootstrap.cc:492] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:38.775887 2024 ts_tablet_manager.cc:1905] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.777441 1181 tablet_bootstrap.cc:654] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.779853 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 90) took 4335 ms. Trace:
I20250112 02:52:38.779937 2003 rpcz_store.cc:276] 0112 02:52:34.444900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.444942 (+ 42us) service_pool.cc:225] Handling call
0112 02:52:38.741723 (+4296781us) ts_tablet_manager.cc:1145] Deleting tablet b8daf4ef3593410eae87672402292be6
0112 02:52:38.741732 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.745018 (+ 3286us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.764692 (+ 19674us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.779839 (+ 15147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4289991}
I20250112 02:52:38.780385 2003 tablet_replica.cc:331] T b61f16f3d601436bb4b32344de28c1a7 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:38.780339 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet b8daf4ef3593410eae87672402292be6 (table default.Table_c0aaadb5e69e46e8983f4d289a44afe2 [id=08299ab225a94e37a67722107a5ff4ce]) successfully deleted
I20250112 02:52:38.780577 2003 raft_consensus.cc:2238] T b61f16f3d601436bb4b32344de28c1a7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.780736 2003 raft_consensus.cc:2267] T b61f16f3d601436bb4b32344de28c1a7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.781715 2003 ts_tablet_manager.cc:1905] T b61f16f3d601436bb4b32344de28c1a7 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.792586 889 tablet_service.cc:1514] Processing DeleteTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.793074 2003 ts_tablet_manager.cc:1918] T b61f16f3d601436bb4b32344de28c1a7 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:38.793172 2003 log.cc:1198] T b61f16f3d601436bb4b32344de28c1a7 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/b61f16f3d601436bb4b32344de28c1a7
I20250112 02:52:38.795192 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.795545 891 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5b3c712c0f448ea9b14ab93dd731ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.795933 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1954e7859f422f9690fe4a9384a8bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.796332 891 tablet_service.cc:1514] Processing DeleteTablet for tablet c29a9ca43e034b43a00207e5847bbee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.796713 891 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e7882e07784bdda18dc1ffd20963c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.797720 899 tablet_service.cc:1514] Processing DeleteTablet for tablet bab4978edea0445fb0456e88e7fb4c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.797900 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c821ea3ca4946c290c17135366e0cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:38786
W20250112 02:52:38.798012 455 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet f3771166b77e4f90aaa116ab1a9dc299 (table default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet f3771166b77e4f90aaa116ab1a9dc299 consensus state has no leader
I20250112 02:52:38.798103 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b1d9fae290ec4281839991f9d40b105c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.798372 2003 ts_tablet_manager.cc:1939] T b61f16f3d601436bb4b32344de28c1a7 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:38.798655 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 06edfca60eed4dd3a2dee7b662b0970c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.798919 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 93) took 4285 ms. Trace:
I20250112 02:52:38.798990 2003 rpcz_store.cc:276] 0112 02:52:34.513729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.514019 (+ 290us) service_pool.cc:225] Handling call
0112 02:52:38.780343 (+4266324us) ts_tablet_manager.cc:1145] Deleting tablet b61f16f3d601436bb4b32344de28c1a7
0112 02:52:38.780353 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.784507 (+ 4154us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.792997 (+ 8490us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.798906 (+ 5909us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4261631}
I20250112 02:52:38.799687 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 91) took 4352 ms. Trace:
I20250112 02:52:38.799746 2003 rpcz_store.cc:276] 0112 02:52:34.495754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.495793 (+ 39us) service_pool.cc:225] Handling call
0112 02:52:38.799588 (+4303795us) ts_tablet_manager.cc:1145] Deleting tablet bab4978edea0445fb0456e88e7fb4c44
0112 02:52:38.799598 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.799674 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4280614}
I20250112 02:52:38.799424 2047 raft_consensus.cc:491] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.800510 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 95) took 4272 ms. Trace:
I20250112 02:52:38.800555 2047 raft_consensus.cc:513] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.800571 2003 rpcz_store.cc:276] 0112 02:52:34.527603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.527657 (+ 54us) service_pool.cc:225] Handling call
0112 02:52:38.800422 (+4272765us) ts_tablet_manager.cc:1145] Deleting tablet 06edfca60eed4dd3a2dee7b662b0970c
0112 02:52:38.800429 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.800498 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4218545}
I20250112 02:52:38.801195 2003 tablet_replica.cc:331] T b9c006281dc949119951c74e23167d7f P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:38.801297 2047 leader_election.cc:290] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.801339 2003 raft_consensus.cc:2238] T b9c006281dc949119951c74e23167d7f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.801440 2003 raft_consensus.cc:2267] T b9c006281dc949119951c74e23167d7f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.801826 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763f77385e404c149c16edabe469c299" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:38.801998 455 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 99407ead9cf547e3bbe78c0d0b327e6a consensus state has no leader
I20250112 02:52:38.802016 1053 raft_consensus.cc:2463] T 763f77385e404c149c16edabe469c299 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:38.803046 714 leader_election.cc:304] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:38.803231 2179 raft_consensus.cc:2798] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250112 02:52:38.803256 455 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 125b12a2f9164653be79d3e6d8313d36 consensus state has no leader
I20250112 02:52:38.803282 2179 raft_consensus.cc:491] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:38.803330 2179 raft_consensus.cc:3054] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.803424 2003 ts_tablet_manager.cc:1905] T b9c006281dc949119951c74e23167d7f P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.804507 1181 tablet_bootstrap.cc:492] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:38.804632 1181 ts_tablet_manager.cc:1397] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20250112 02:52:38.805243 1181 raft_consensus.cc:357] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.805433 1181 raft_consensus.cc:383] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.805498 1181 raft_consensus.cc:738] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.805513 2179 raft_consensus.cc:513] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.805701 1181 consensus_queue.cc:260] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.806032 2179 leader_election.cc:290] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.806197 1181 ts_tablet_manager.cc:1428] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:38.806241 2496 raft_consensus.cc:491] T 7ad9c47fcb534619949971796b60eba2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.806327 2496 raft_consensus.cc:513] T 7ad9c47fcb534619949971796b60eba2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.806834 2496 leader_election.cc:290] T 7ad9c47fcb534619949971796b60eba2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:38.807657 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763f77385e404c149c16edabe469c299" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:38.807824 1049 raft_consensus.cc:3054] T 763f77385e404c149c16edabe469c299 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.808033 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763f77385e404c149c16edabe469c299" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:38.808176 910 raft_consensus.cc:3054] T 763f77385e404c149c16edabe469c299 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.810089 1049 raft_consensus.cc:2463] T 763f77385e404c149c16edabe469c299 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:38.810478 910 raft_consensus.cc:2463] T 763f77385e404c149c16edabe469c299 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:38.810851 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad9c47fcb534619949971796b60eba2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:38.811250 854 leader_election.cc:343] T 7ad9c47fcb534619949971796b60eba2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.811250 714 leader_election.cc:304] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:38.812947 892 tablet_service.cc:1514] Processing DeleteTablet for tablet f7cd0927c6a4401cbe69f55d4db9963a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.813196 887 tablet_service.cc:1514] Processing DeleteTablet for tablet d659c5a4f7a446c6aaf2283f65b45106 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.813261 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d0f67141dc46b6a87ca0ad080d5bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.813462 892 tablet_service.cc:1514] Processing DeleteTablet for tablet cc62dcbf67e84e36ae1fbe127ccd4879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.813683 892 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d411f906b4d4eb68343bdcc493e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.813849 893 tablet_service.cc:1514] Processing DeleteTablet for tablet cbe30f9860e2492b90e8cedf8ff9daba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.813988 892 tablet_service.cc:1514] Processing DeleteTablet for tablet daef3a58605e40978f7389698b22265d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.814038 895 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ffdc237f8640f0b378809f93fc6292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.813994 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 54215c55966a46e1ab96e352c4e3764e with delete_type TABLET_DATA_DELETED (replaced by tablet a470f84d50df4cf0b4a1e5d248b4c32a) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.814311 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c04f1a2cb04950a9f6170e252c19b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.814567 2024 ts_tablet_manager.cc:1918] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:38.814579 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b351c563d749df806ab5d0331a4ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.814662 2024 log.cc:1198] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/f234256b343f447dadfc3d1936a824ee
I20250112 02:52:38.814729 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1e3af2883341e9b957f6ed748e604d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.814889 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a8a178bafcb457f9b0e9587df560493 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.814975 2024 ts_tablet_manager.cc:1939] T f234256b343f447dadfc3d1936a824ee P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:38.815044 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 13578542e9e741e8817e0ee63b10f1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.815466 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad9c47fcb534619949971796b60eba2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:38.815786 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 04f135b226774285a077e5806131d91b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.815780 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 103) took 4278 ms. Trace:
I20250112 02:52:38.815863 2024 rpcz_store.cc:276] 0112 02:52:34.537723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.541808 (+ 4085us) service_pool.cc:225] Handling call
0112 02:52:38.775149 (+4233341us) ts_tablet_manager.cc:1145] Deleting tablet f234256b343f447dadfc3d1936a824ee
0112 02:52:38.775157 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.778552 (+ 3395us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.814464 (+ 35912us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.815763 (+ 1299us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4233129}
I20250112 02:52:38.816246 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 161fe26b9c0e4543b7aeb0021506d72a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.816397 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 416fb6820cd54e118950c68d381f51b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
W20250112 02:52:38.816581 855 leader_election.cc:343] T 7ad9c47fcb534619949971796b60eba2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.816676 855 leader_election.cc:304] T 7ad9c47fcb534619949971796b60eba2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:38.817577 2496 raft_consensus.cc:2743] T 7ad9c47fcb534619949971796b60eba2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:38.817829 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f4bd2df0ce45dabc55373f1154d3ac with delete_type TABLET_DATA_DELETED (replaced by tablet f559dedfb7854cf890bef362c50b4213) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.818076 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet b61f16f3d601436bb4b32344de28c1a7 (table default.Table_dafd4ddc8ab34adfa999f9340216f765 [id=da4d2d7725904d5bb4a6bde668df00ef]) successfully deleted
W20250112 02:52:38.818320 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet bab4978edea0445fb0456e88e7fb4c44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bab4978edea0445fb0456e88e7fb4c44 already in progress: creating tablet
W20250112 02:52:38.818508 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 06edfca60eed4dd3a2dee7b662b0970c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06edfca60eed4dd3a2dee7b662b0970c already in progress: creating tablet
I20250112 02:52:38.818783 2179 raft_consensus.cc:2798] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:38.818888 2179 raft_consensus.cc:695] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:38.819379 2179 consensus_queue.cc:237] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.820019 2003 ts_tablet_manager.cc:1918] T b9c006281dc949119951c74e23167d7f P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:38.820107 2003 log.cc:1198] T b9c006281dc949119951c74e23167d7f P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/b9c006281dc949119951c74e23167d7f
I20250112 02:52:38.820351 2003 ts_tablet_manager.cc:1939] T b9c006281dc949119951c74e23167d7f P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:38.820513 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763f77385e404c149c16edabe469c299" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:38.820606 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 94) took 4301 ms. Trace:
I20250112 02:52:38.820667 2003 rpcz_store.cc:276] 0112 02:52:34.518819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.518973 (+ 154us) service_pool.cc:225] Handling call
0112 02:52:38.801163 (+4282190us) ts_tablet_manager.cc:1145] Deleting tablet b9c006281dc949119951c74e23167d7f
0112 02:52:38.801169 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.816758 (+ 15589us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.819955 (+ 3197us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.820593 (+ 638us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4219138}
I20250112 02:52:38.820782 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 104) took 4282 ms. Trace:
I20250112 02:52:38.820865 2024 rpcz_store.cc:276] 0112 02:52:34.537835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.541905 (+ 4070us) service_pool.cc:225] Handling call
0112 02:52:38.820627 (+4278722us) ts_tablet_manager.cc:1145] Deleting tablet 06edfca60eed4dd3a2dee7b662b0970c
0112 02:52:38.820637 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.820762 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4278584}
I20250112 02:52:38.821027 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet b9c006281dc949119951c74e23167d7f (table default.Table_1a6ba73a53b14addbacc6a2ff0711256 [id=fe95e997b21346a5b36675a9c4d25a06]) successfully deleted
I20250112 02:52:38.821209 917 raft_consensus.cc:2371] T 763f77385e404c149c16edabe469c299 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1. Re-sending same reply.
I20250112 02:52:38.822032 1181 tablet_bootstrap.cc:492] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:38.823388 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 49bb6d20cfa0453e90e048825114525e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.824278 1181 tablet_bootstrap.cc:654] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:38.825016 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 06edfca60eed4dd3a2dee7b662b0970c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06edfca60eed4dd3a2dee7b662b0970c already in progress: creating tablet
I20250112 02:52:38.825142 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 105) took 4287 ms. Trace:
I20250112 02:52:38.825681 2024 rpcz_store.cc:276] 0112 02:52:34.537924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.542047 (+ 4123us) service_pool.cc:225] Handling call
0112 02:52:38.825054 (+4283007us) ts_tablet_manager.cc:1145] Deleting tablet f776e2a6ec704cf9b64483bc7d5b1dff
0112 02:52:38.825062 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.825129 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4282802}
W20250112 02:52:38.826431 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f776e2a6ec704cf9b64483bc7d5b1dff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f776e2a6ec704cf9b64483bc7d5b1dff already in progress: creating tablet
I20250112 02:52:38.826512 2024 tablet_replica.cc:331] T c8a7281c0c09442ebd0cc13b8496c86e P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:38.826679 2024 raft_consensus.cc:2238] T c8a7281c0c09442ebd0cc13b8496c86e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.826829 2024 raft_consensus.cc:2267] T c8a7281c0c09442ebd0cc13b8496c86e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.827335 2024 ts_tablet_manager.cc:1905] T c8a7281c0c09442ebd0cc13b8496c86e P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.828147 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 49bb6d20cfa0453e90e048825114525e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.828632 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 41e166b3f8414ababbd4a3f07fb90c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.829023 2047 raft_consensus.cc:491] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 02:52:38.829353 455 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 61eebc5f22664712af6e74e82809ca87 consensus state has no leader
I20250112 02:52:38.829427 2228 raft_consensus.cc:491] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.829492 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f4e4f1e451b48dc9cdd780300f6888c with delete_type TABLET_DATA_DELETED (replaced by tablet 2052b4536f9745f0a785f330d706c1cf) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.829548 2228 raft_consensus.cc:513] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:38.829910 891 tablet_service.cc:1514] Processing DeleteTablet for tablet fe614eaeb793473d9b2da206b7345562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.830125 2228 leader_election.cc:290] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.830494 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 70715db0394d46d9b6b289160b00be7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.831113 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 87759e8b99b04e628badc8f60cb38bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.831632 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c639a2bcd4740e4be7b2b162004669a with delete_type TABLET_DATA_DELETED (replaced by tablet 5d0c3191973a477e949fbf9dfcca963f) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.832190 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 61593969d2234ce3a6089027b067d922 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.832767 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ee98f055024429eafaa29bb961cab87 with delete_type TABLET_DATA_DELETED (replaced by tablet 99407ead9cf547e3bbe78c0d0b327e6a) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.832908 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2f1f1a76e5f4308bf56dd647f02c6d8" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:38.833108 917 raft_consensus.cc:2463] T e2f1f1a76e5f4308bf56dd647f02c6d8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:38.833335 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d84027ad6d40a5be5afcdd854431a5 with delete_type TABLET_DATA_DELETED (replaced by tablet fbe9e9624ede40c29ea852b2b5494965) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.833506 989 leader_election.cc:304] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:38.833781 2228 raft_consensus.cc:2798] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:38.833882 2228 raft_consensus.cc:491] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:38.833912 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a34d03a80ae402db6607261a5c40b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.833952 2228 raft_consensus.cc:3054] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.834309 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af55dab202444358ea1a0674d37946d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.834913 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 058a9ca089cd4967887b0a1127c7d03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.835354 887 tablet_service.cc:1514] Processing DeleteTablet for tablet d45fce86e02f46d4a57af718fa2a62c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 5804fc4130f84c69832c1022b7bd7fa5) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.835927 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a18132549374218893f35a4e4226b72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.836381 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 003a3b4705e943c6a900762ec11ddd47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.836455 2228 raft_consensus.cc:513] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:38.836951 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 2730634a5cfd4c2fb5c4e30100c556d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.837456 887 tablet_service.cc:1514] Processing DeleteTablet for tablet b61f16f3d601436bb4b32344de28c1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.837647 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2f1f1a76e5f4308bf56dd647f02c6d8" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:38.837807 910 raft_consensus.cc:3054] T e2f1f1a76e5f4308bf56dd647f02c6d8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.840553 910 raft_consensus.cc:2463] T e2f1f1a76e5f4308bf56dd647f02c6d8 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:38.841157 989 leader_election.cc:304] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:38.841415 2326 raft_consensus.cc:2798] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:38.841590 2326 raft_consensus.cc:695] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:38.841871 2326 consensus_queue.cc:237] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:38.842167 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet f776e2a6ec704cf9b64483bc7d5b1dff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f776e2a6ec704cf9b64483bc7d5b1dff already in progress: creating tablet
W20250112 02:52:38.842609 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b because the tablet was not found. No further retry: Not found: Tablet not found: 4348fe4c32704bf7b3a8d21ff3b8ac1b
I20250112 02:52:38.842839 2562 tablet_replica.cc:331] T bd5b3c712c0f448ea9b14ab93dd731ab P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:38.842989 2496 raft_consensus.cc:491] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.843062 2562 raft_consensus.cc:2238] T bd5b3c712c0f448ea9b14ab93dd731ab P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.843200 2562 raft_consensus.cc:2267] T bd5b3c712c0f448ea9b14ab93dd731ab P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.843111 2496 raft_consensus.cc:513] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.843791 2496 leader_election.cc:290] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:38.844549 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f3be6bcbe8a45f0a8076b4842d90dc2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:38.844967 2024 ts_tablet_manager.cc:1918] T c8a7281c0c09442ebd0cc13b8496c86e P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:38.848871 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f3be6bcbe8a45f0a8076b4842d90dc2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:38.849709 855 leader_election.cc:343] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.849807 2399 raft_consensus.cc:491] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.849907 2399 raft_consensus.cc:513] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:38.850466 2399 leader_election.cc:290] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:38.850580 2399 raft_consensus.cc:491] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.850672 2399 raft_consensus.cc:513] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.851151 2399 leader_election.cc:290] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:38.852145 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d662e0ce13f447595e5f92b94de3018" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:38.852993 854 leader_election.cc:343] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.855000 2228 leader_election.cc:290] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:38.855017 854 leader_election.cc:343] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.855089 854 leader_election.cc:304] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:38.855291 2024 log.cc:1198] T c8a7281c0c09442ebd0cc13b8496c86e P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/c8a7281c0c09442ebd0cc13b8496c86e
I20250112 02:52:38.855269 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b351c563d749df806ab5d0331a4ef2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:38.855378 1173 tablet_bootstrap.cc:492] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:38.855389 2496 raft_consensus.cc:2743] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:38.855710 854 leader_election.cc:343] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.855888 2024 ts_tablet_manager.cc:1939] T c8a7281c0c09442ebd0cc13b8496c86e P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:38.856038 2003 tablet_replica.cc:331] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:38.856236 2003 raft_consensus.cc:2238] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:38.856413 2003 pending_rounds.cc:62] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:38.856452 2003 pending_rounds.cc:69] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319866662154240 op_type: NO_OP noop_request { }
I20250112 02:52:38.856536 2003 raft_consensus.cc:2883] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:38.856617 2003 raft_consensus.cc:2267] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.856818 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 94) took 4321 ms. Trace:
I20250112 02:52:38.856897 1173 ts_tablet_manager.cc:1397] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.094s user 0.005s sys 0.014s
I20250112 02:52:38.856941 2024 rpcz_store.cc:276] 0112 02:52:34.535379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.540875 (+ 5496us) service_pool.cc:225] Handling call
0112 02:52:38.826476 (+4285601us) ts_tablet_manager.cc:1145] Deleting tablet c8a7281c0c09442ebd0cc13b8496c86e
0112 02:52:38.826483 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.830147 (+ 3664us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.844881 (+ 14734us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.856771 (+ 11890us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4284197}
I20250112 02:52:38.857424 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet c8a7281c0c09442ebd0cc13b8496c86e (table default.Table_105e5d1a8ef846ffbcb1bdbe01bbae55 [id=082744a588e64fe8bafe0bc442876fc9]) successfully deleted
I20250112 02:52:38.857582 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 87) took 4322 ms. Trace:
I20250112 02:52:38.857743 2024 rpcz_store.cc:276] 0112 02:52:34.534672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.539401 (+ 4729us) service_pool.cc:225] Handling call
0112 02:52:38.857421 (+4318020us) ts_tablet_manager.cc:1145] Deleting tablet b2ccb88108f24e36ba4b7ebf36398950
0112 02:52:38.857429 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.857474 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4315074}
I20250112 02:52:38.857987 1173 raft_consensus.cc:357] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:38.858134 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b2ccb88108f24e36ba4b7ebf36398950 because the tablet was not found. No further retry: Not found: Tablet not found: b2ccb88108f24e36ba4b7ebf36398950
I20250112 02:52:38.858155 1173 raft_consensus.cc:383] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.858240 1173 raft_consensus.cc:738] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.858424 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 107) took 4320 ms. Trace:
I20250112 02:52:38.858712 2024 rpcz_store.cc:276] 0112 02:52:34.538118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.542268 (+ 4150us) service_pool.cc:225] Handling call
0112 02:52:38.858114 (+4315846us) ts_tablet_manager.cc:1145] Deleting tablet 8c821ea3ca4946c290c17135366e0cac
0112 02:52:38.858184 (+ 70us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.858365 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4315677}
I20250112 02:52:38.858425 1173 consensus_queue.cc:260] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:38.859110 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 8c821ea3ca4946c290c17135366e0cac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c821ea3ca4946c290c17135366e0cac already in progress: creating tablet
I20250112 02:52:38.859508 1173 ts_tablet_manager.cc:1428] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:38.859609 2003 ts_tablet_manager.cc:1905] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.859921 1173 tablet_bootstrap.cc:492] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:38.862113 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5b3c712c0f448ea9b14ab93dd731ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.862299 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 106) took 4324 ms. Trace:
I20250112 02:52:38.862385 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet b1d9fae290ec4281839991f9d40b105c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.862644 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet bab4978edea0445fb0456e88e7fb4c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.863014 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c821ea3ca4946c290c17135366e0cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.863363 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.863706 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 06edfca60eed4dd3a2dee7b662b0970c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.863952 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet f234256b343f447dadfc3d1936a824ee with delete_type TABLET_DATA_DELETED (replaced by tablet 82b351c563d749df806ab5d0331a4ef2) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.864274 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet b61f16f3d601436bb4b32344de28c1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.864460 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet a12c5d58e6c246c983e8a533906ce77d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.864888 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 2730634a5cfd4c2fb5c4e30100c556d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.865283 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet d45fce86e02f46d4a57af718fa2a62c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 5804fc4130f84c69832c1022b7bd7fa5) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.865281 1173 tablet_bootstrap.cc:654] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.866019 1181 tablet_bootstrap.cc:492] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:38.866132 1181 ts_tablet_manager.cc:1397] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.007s
I20250112 02:52:38.866685 1181 raft_consensus.cc:357] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.866842 1181 raft_consensus.cc:383] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.866905 1181 raft_consensus.cc:738] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.862569 2024 rpcz_store.cc:276] 0112 02:52:34.538024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.542062 (+ 4038us) service_pool.cc:225] Handling call
0112 02:52:38.859756 (+4317694us) ts_tablet_manager.cc:1145] Deleting tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b
0112 02:52:38.862007 (+ 2251us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.862080 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4317295}
I20250112 02:52:38.867815 2024 tablet_replica.cc:331] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
W20250112 02:52:38.867837 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b already in progress: creating tablet
I20250112 02:52:38.868119 2024 raft_consensus.cc:2238] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:38.869706 1181 consensus_queue.cc:260] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:38.868860 989 consensus_peers.cc:487] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 416fb6820cd54e118950c68d381f51b2. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:38.870811 2562 ts_tablet_manager.cc:1905] T bd5b3c712c0f448ea9b14ab93dd731ab P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.872269 1181 ts_tablet_manager.cc:1428] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20250112 02:52:38.872511 1181 tablet_bootstrap.cc:492] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:38.873247 1173 tablet_bootstrap.cc:492] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:38.873379 1173 ts_tablet_manager.cc:1397] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20250112 02:52:38.873879 1181 tablet_bootstrap.cc:654] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.874013 1173 raft_consensus.cc:357] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.874200 1173 raft_consensus.cc:383] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.874280 1173 raft_consensus.cc:738] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.874703 2024 raft_consensus.cc:2267] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.875240 2024 ts_tablet_manager.cc:1905] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.881034 887 tablet_service.cc:1514] Processing DeleteTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.881903 751 tablet_service.cc:1514] Processing DeleteTablet for tablet b9199b0d2b924a79938a979df4196bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.882336 887 tablet_service.cc:1514] Processing DeleteTablet for tablet b9c006281dc949119951c74e23167d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.883059 2317 raft_consensus.cc:491] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.883184 2317 raft_consensus.cc:513] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.883334 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb76746b3a846c3aba531c04bf63966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.883919 756 tablet_service.cc:1514] Processing DeleteTablet for tablet b9c006281dc949119951c74e23167d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.884090 754 tablet_service.cc:1514] Processing DeleteTablet for tablet f05c8426ea3d4fe89fbe6c88c688019f with delete_type TABLET_DATA_DELETED (replaced by tablet c5079f87652a4ea58b366d1f9448fd4e) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.884912 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 73136be25c79491784d3ea71aa9b27f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:38.885051 713 leader_election.cc:343] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 003a3b4705e943c6a900762ec11ddd47
I20250112 02:52:38.885677 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b351c563d749df806ab5d0331a4ef2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:38.886085 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2f1f1a76e5f4308bf56dd647f02c6d8" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:38.886226 775 raft_consensus.cc:2463] T e2f1f1a76e5f4308bf56dd647f02c6d8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:38.829131 2047 raft_consensus.cc:513] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.886713 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 2730634a5cfd4c2fb5c4e30100c556d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.886955 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 06edfca60eed4dd3a2dee7b662b0970c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.883806 2317 leader_election.cc:290] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.888229 1181 tablet_bootstrap.cc:492] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:38.888337 1181 ts_tablet_manager.cc:1397] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20250112 02:52:38.889003 1181 raft_consensus.cc:357] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.889196 1181 raft_consensus.cc:383] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.889243 1181 raft_consensus.cc:738] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.891147 2525 tablet.cc:1721] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:38.892009 2326 raft_consensus.cc:491] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.892311 2522 tablet.cc:1721] T 5cc414d6420c41839621ab858067548b P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:38.892246 2326 raft_consensus.cc:513] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.892997 2326 leader_election.cc:290] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.893550 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795ac7eaf81d48bb926865d9ebb388b6" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:38.893700 917 raft_consensus.cc:2463] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:38.894090 989 leader_election.cc:304] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:38.894723 2326 raft_consensus.cc:2798] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:38.894836 2326 raft_consensus.cc:491] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:38.894908 2326 raft_consensus.cc:3054] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.895053 1181 consensus_queue.cc:260] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.895567 1181 ts_tablet_manager.cc:1428] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250112 02:52:38.895816 1181 tablet_bootstrap.cc:492] T d63d411f906b4d4eb68343bdcc493e05 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:38.897876 2326 raft_consensus.cc:513] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.899142 1181 tablet_bootstrap.cc:654] T d63d411f906b4d4eb68343bdcc493e05 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.899636 2326 leader_election.cc:290] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.899952 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795ac7eaf81d48bb926865d9ebb388b6" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:38.900115 917 raft_consensus.cc:3054] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.901685 1173 consensus_queue.cc:260] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.902637 1173 ts_tablet_manager.cc:1428] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20250112 02:52:38.902931 1173 tablet_bootstrap.cc:492] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:38.902954 917 raft_consensus.cc:2463] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:38.903344 989 leader_election.cc:304] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:38.904139 2326 raft_consensus.cc:2798] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:38.904337 2326 raft_consensus.cc:695] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:38.904624 2326 consensus_queue.cc:237] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.905445 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 70715db0394d46d9b6b289160b00be7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:38.907194 455 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:58959 (request call id 239) took 1753 ms (client timeout 1000 ms). Trace:
I20250112 02:52:38.909258 2024 ts_tablet_manager.cc:1918] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:38.924388 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.927431 760 tablet_service.cc:1514] Processing DeleteTablet for tablet b61f16f3d601436bb4b32344de28c1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.928391 2003 ts_tablet_manager.cc:1918] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:38.928516 2003 log.cc:1198] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/f234256b343f447dadfc3d1936a824ee
I20250112 02:52:38.928877 2003 ts_tablet_manager.cc:1939] T f234256b343f447dadfc3d1936a824ee P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:38.929225 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 96) took 4369 ms. Trace:
I20250112 02:52:38.929517 2003 rpcz_store.cc:276] 0112 02:52:34.559497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.559540 (+ 43us) service_pool.cc:225] Handling call
0112 02:52:38.855986 (+4296446us) ts_tablet_manager.cc:1145] Deleting tablet f234256b343f447dadfc3d1936a824ee
0112 02:52:38.855997 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.900300 (+ 44303us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.928305 (+ 28005us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.929211 (+ 906us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4273785}
I20250112 02:52:38.930200 2445 consensus_queue.cc:1035] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.098s
I20250112 02:52:38.914314 1173 tablet_bootstrap.cc:654] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.918241 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af55dab202444358ea1a0674d37946d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.920370 2326 raft_consensus.cc:491] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:38.921380 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet e9fc9a4f1e8944abb76dad6c2ad53d35 with delete_type TABLET_DATA_DELETED (replaced by tablet 5cc414d6420c41839621ab858067548b) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.923439 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet aba1744138fe42ae9d097db01cac083d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.926862 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 015412a5fa304453bd8d1be1c3cd7a46 with delete_type TABLET_DATA_DELETED (replaced by tablet f924459d1150472aafb325a11825db23) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.929405 2228 consensus_queue.cc:1035] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:38.931780 2326 raft_consensus.cc:513] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.929636 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d411f906b4d4eb68343bdcc493e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.933161 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4ef0220dde43569452379639759db5" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:38.933360 910 raft_consensus.cc:2463] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:38.933766 2326 leader_election.cc:290] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.934026 989 leader_election.cc:304] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:38.934329 2326 raft_consensus.cc:2798] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:38.934422 2326 raft_consensus.cc:491] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:38.934540 2326 raft_consensus.cc:3054] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.937295 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet d659c5a4f7a446c6aaf2283f65b45106 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.937707 2326 raft_consensus.cc:513] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.938515 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d0f67141dc46b6a87ca0ad080d5bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.938643 2326 leader_election.cc:290] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.938885 2024 log.cc:1198] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/9d5b3df88a0c45dba879b3d506710614
I20250112 02:52:38.938957 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4ef0220dde43569452379639759db5" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:38.939126 910 raft_consensus.cc:3054] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.939324 2024 ts_tablet_manager.cc:1939] T 9d5b3df88a0c45dba879b3d506710614 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:38.941535 910 raft_consensus.cc:2463] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:38.942304 989 leader_election.cc:304] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:38.942565 2326 raft_consensus.cc:2798] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:38.942842 2326 raft_consensus.cc:695] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:38.943157 2326 consensus_queue.cc:237] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.944718 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f22a7b1b804661af3b7e785a833739 with delete_type TABLET_DATA_DELETED (replaced by tablet 3d662e0ce13f447595e5f92b94de3018) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.945171 1173 tablet_bootstrap.cc:492] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:38.945308 1173 ts_tablet_manager.cc:1397] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.002s
I20250112 02:52:38.946166 1173 raft_consensus.cc:357] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.946381 1173 raft_consensus.cc:383] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.946482 1173 raft_consensus.cc:738] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.946708 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 416fb6820cd54e118950c68d381f51b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.946785 1173 consensus_queue.cc:260] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.947584 1173 ts_tablet_manager.cc:1428] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250112 02:52:38.946964 455 rpcz_store.cc:269] 0112 02:52:37.153602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:37.775934 (+622332us) service_pool.cc:225] Handling call
0112 02:52:38.907152 (+1131218us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:38.048346 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:38.048447 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:38.048454 (+ 7us) write_op.cc:435] Acquired schema lock
0112 02:52:38.048458 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:38.048565 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:38.048579 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:38.048586 (+ 7us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:38.048659 (+ 73us) tablet.cc:698] Row locks acquired
0112 02:52:38.048662 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:38.048694 (+ 32us) write_op.cc:270] Start()
0112 02:52:38.048727 (+ 33us) write_op.cc:276] Timestamp: P: 1736650358048527 usec, L: 0
0112 02:52:38.048732 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:38.048911 (+ 179us) log.cc:844] Serialized 527 byte log entry
0112 02:52:38.053555 (+ 4644us) op_driver.cc:464] REPLICATION: finished
0112 02:52:38.053677 (+ 122us) write_op.cc:301] APPLY: starting
0112 02:52:38.053689 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:38.053728 (+ 39us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:38.053764 (+ 36us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:38.053808 (+ 44us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:38.053838 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:38.053848 (+ 10us) write_op.cc:312] APPLY: finished
0112 02:52:38.053913 (+ 65us) log.cc:844] Serialized 23 byte log entry
0112 02:52:38.053979 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.053997 (+ 18us) write_op.cc:454] Released schema lock
0112 02:52:38.054005 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:38.054070 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.054073 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:38.289245 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:38.289324 (+ 79us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:38.289329 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:38.289333 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:38.289395 (+ 62us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:38.289403 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:38.289407 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:38.289445 (+ 38us) tablet.cc:698] Row locks acquired
0112 02:52:38.289447 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:38.289470 (+ 23us) write_op.cc:270] Start()
0112 02:52:38.289490 (+ 20us) write_op.cc:276] Timestamp: P: 1736650358289419 usec, L: 0
0112 02:52:38.289493 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:38.289603 (+ 110us) log.cc:844] Serialized 356 byte log entry
0112 02:52:38.294904 (+ 5301us) op_driver.cc:464] REPLICATION: finished
0112 02:52:38.295000 (+ 96us) write_op.cc:301] APPLY: starting
0112 02:52:38.295011 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:38.295036 (+ 25us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:38.295055 (+ 19us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:38.295085 (+ 30us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:38.295107 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:38.295114 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:38.295158 (+ 44us) log.cc:844] Serialized 23 byte log entry
0112 02:52:38.295200 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.295213 (+ 13us) write_op.cc:454] Released schema lock
0112 02:52:38.295218 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:38.295281 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.295285 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:38.723064 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:38.723141 (+ 77us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:38.723145 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:38.723147 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:38.723217 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:38.723228 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:38.723234 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:38.723281 (+ 47us) tablet.cc:698] Row locks acquired
0112 02:52:38.723284 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:38.723307 (+ 23us) write_op.cc:270] Start()
0112 02:52:38.723326 (+ 19us) write_op.cc:276] Timestamp: P: 1736650358723270 usec, L: 0
0112 02:52:38.723329 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:38.723441 (+ 112us) log.cc:844] Serialized 356 byte log entry
0112 02:52:38.725447 (+ 2006us) op_driver.cc:464] REPLICATION: finished
0112 02:52:38.725531 (+ 84us) write_op.cc:301] APPLY: starting
0112 02:52:38.725542 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:38.725568 (+ 26us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:38.725589 (+ 21us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:38.725624 (+ 35us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:38.725646 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:38.725653 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:38.725699 (+ 46us) log.cc:844] Serialized 23 byte log entry
0112 02:52:38.725754 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.725766 (+ 12us) write_op.cc:454] Released schema lock
0112 02:52:38.725771 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:38.725829 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.725832 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:38.755521 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:38.755616 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:38.755621 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:38.755623 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:38.755713 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:38.755723 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:38.755727 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:38.755782 (+ 55us) tablet.cc:698] Row locks acquired
0112 02:52:38.755784 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:38.755812 (+ 28us) write_op.cc:270] Start()
0112 02:52:38.755840 (+ 28us) write_op.cc:276] Timestamp: P: 1736650358755766 usec, L: 0
0112 02:52:38.755843 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:38.756000 (+ 157us) log.cc:844] Serialized 356 byte log entry
0112 02:52:38.758751 (+ 2751us) op_driver.cc:464] REPLICATION: finished
0112 02:52:38.758854 (+ 103us) write_op.cc:301] APPLY: starting
0112 02:52:38.758867 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:38.758901 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:38.758949 (+ 48us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:38.758992 (+ 43us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:38.759025 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:38.759033 (+ 8us) write_op.cc:312] APPLY: finished
0112 02:52:38.759086 (+ 53us) log.cc:844] Serialized 23 byte log entry
0112 02:52:38.759136 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.759151 (+ 15us) write_op.cc:454] Released schema lock
0112 02:52:38.759158 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:38.759226 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.759231 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":63325,"reported_tablets":198,"child_traces":[["op",{"apply.queue_time_us":79,"apply.run_cpu_time_us":386,"apply.run_wall_time_us":387,"num_ops":1,"prepare.queue_time_us":1653,"prepare.run_cpu_time_us":660,"prepare.run_wall_time_us":660,"raft.queue_time_us":435,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":240,"replication_time_us":4793}],["op",{"apply.queue_time_us":62,"apply.run_cpu_time_us":274,"apply.run_wall_time_us":274,"num_ops":1,"prepare.queue_time_us":128,"prepare.run_cpu_time_us":425,"prepare.run_wall_time_us":425,"raft.queue_time_us":88,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":170,"replication_time_us":5390}],["op",{"apply.queue_time_us":47,"apply.run_cpu_time_us":290,"apply.run_wall_time_us":289,"num_ops":1,"prepare.queue_time_us":38,"prepare.run_cpu_time_us":474,"prepare.run_wall_time_us":481,"raft.queue_time_us":47,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":152,"replication_time_us":2095}],["op",{"apply.queue_time_us":64,"apply.run_cpu_time_us":362,"apply.run_wall_time_us":361,"num_ops":1,"prepare.queue_time_us":44,"prepare.run_cpu_time_us":571,"prepare.run_wall_time_us":572,"raft.queue_time_us":282,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":220,"replication_time_us":2876}]]}
I20250112 02:52:38.950582 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1954e7859f422f9690fe4a9384a8bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.952545 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 161fe26b9c0e4543b7aeb0021506d72a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.952948 1173 tablet_bootstrap.cc:492] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:38.953560 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet afa2f8aabe5e42ccbc15b193fdc03563 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.953797 2047 leader_election.cc:290] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:38.954183 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd498ccce484a89855c982762086890 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.954208 826 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:38.954205 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795ac7eaf81d48bb926865d9ebb388b6" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:38.954308 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 97) took 4394 ms. Trace:
I20250112 02:52:38.954378 910 raft_consensus.cc:2388] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 289afc116aa74408aa5bff74e11c50d8 in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
I20250112 02:52:38.954383 756 tablet_service.cc:1514] Processing DeleteTablet for tablet cc62dcbf67e84e36ae1fbe127ccd4879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.954370 2003 rpcz_store.cc:276] 0112 02:52:34.559634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.559670 (+ 36us) service_pool.cc:225] Handling call
0112 02:52:38.954186 (+4394516us) ts_tablet_manager.cc:1145] Deleting tablet 8c821ea3ca4946c290c17135366e0cac
0112 02:52:38.954196 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.954293 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4370931}
I20250112 02:52:38.954780 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 98) took 4395 ms. Trace:
I20250112 02:52:38.954855 2003 rpcz_store.cc:276] 0112 02:52:34.559763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.559804 (+ 41us) service_pool.cc:225] Handling call
0112 02:52:38.954664 (+4394860us) ts_tablet_manager.cc:1145] Deleting tablet b1d9fae290ec4281839991f9d40b105c
0112 02:52:38.954674 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.954750 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4371237}
I20250112 02:52:38.955160 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 99) took 4395 ms. Trace:
I20250112 02:52:38.955245 2003 rpcz_store.cc:276] 0112 02:52:34.559889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.559960 (+ 71us) service_pool.cc:225] Handling call
0112 02:52:38.955054 (+4395094us) ts_tablet_manager.cc:1145] Deleting tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b
0112 02:52:38.955062 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.955142 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4371443}
I20250112 02:52:38.955405 1173 tablet_bootstrap.cc:654] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.955582 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 100) took 4395 ms. Trace:
I20250112 02:52:38.955665 2003 rpcz_store.cc:276] 0112 02:52:34.560019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.560066 (+ 47us) service_pool.cc:225] Handling call
0112 02:52:38.955486 (+4395420us) ts_tablet_manager.cc:1145] Deleting tablet f776e2a6ec704cf9b64483bc7d5b1dff
0112 02:52:38.955494 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.955567 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4371849}
I20250112 02:52:38.955847 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet b9199b0d2b924a79938a979df4196bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.955991 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 102) took 4395 ms. Trace:
I20250112 02:52:38.956079 2003 rpcz_store.cc:276] 0112 02:52:34.560330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.560958 (+ 628us) service_pool.cc:225] Handling call
0112 02:52:38.955898 (+4394940us) ts_tablet_manager.cc:1145] Deleting tablet 62f4bd2df0ce45dabc55373f1154d3ac
0112 02:52:38.955906 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.955976 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4368182}
I20250112 02:52:38.956391 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 92) took 4446 ms. Trace:
I20250112 02:52:38.956482 2003 rpcz_store.cc:276] 0112 02:52:34.509478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.513985 (+ 4507us) service_pool.cc:225] Handling call
0112 02:52:38.956280 (+4442295us) ts_tablet_manager.cc:1145] Deleting tablet 058a9ca089cd4967887b0a1127c7d03f
0112 02:52:38.956290 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.956375 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4358643}
I20250112 02:52:38.956835 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 104) took 4299 ms. Trace:
I20250112 02:52:38.956916 2003 rpcz_store.cc:276] 0112 02:52:34.657014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.657062 (+ 48us) service_pool.cc:225] Handling call
0112 02:52:38.956716 (+4299654us) ts_tablet_manager.cc:1145] Deleting tablet a8e7882e07784bdda18dc1ffd20963c8
0112 02:52:38.956726 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.956821 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4255702}
I20250112 02:52:38.957170 2003 tablet_replica.cc:331] T 8ee98f055024429eafaa29bb961cab87 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:38.957398 1173 tablet_bootstrap.cc:492] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:38.957357 743 tablet_service.cc:1514] Processing DeleteTablet for tablet bab4978edea0445fb0456e88e7fb4c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.957535 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1e3af2883341e9b957f6ed748e604d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.957612 1173 ts_tablet_manager.cc:1397] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250112 02:52:38.957755 751 tablet_service.cc:1514] Processing DeleteTablet for tablet c7c530aa0aa64e47ae27d61f341d2bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.957978 751 tablet_service.cc:1514] Processing DeleteTablet for tablet fe614eaeb793473d9b2da206b7345562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.958210 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 41e166b3f8414ababbd4a3f07fb90c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.957365 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 015412a5fa304453bd8d1be1c3cd7a46 with delete_type TABLET_DATA_DELETED (replaced by tablet f924459d1150472aafb325a11825db23) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.958495 1173 raft_consensus.cc:357] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:38.958698 1173 raft_consensus.cc:383] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.958757 1173 raft_consensus.cc:738] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.958753 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f4e4f1e451b48dc9cdd780300f6888c with delete_type TABLET_DATA_DELETED (replaced by tablet 2052b4536f9745f0a785f330d706c1cf) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.959040 1173 consensus_queue.cc:260] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:38.960433 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a12b225131b464494a9f02133d37aea with delete_type TABLET_DATA_DELETED (replaced by tablet d45fce86e02f46d4a57af718fa2a62c8) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.960613 755 tablet_service.cc:1514] Processing DeleteTablet for tablet ac716c6da01941a989754aac1ef430d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.960733 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 61593969d2234ce3a6089027b067d922 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.960103 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c686fdc8896463c81c9f7c5f53a3da2 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c03b5e612d64314b210c11686fdfa2a) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.961041 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a8a178bafcb457f9b0e9587df560493 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.962145 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 64707ee432fb4e96a3bbf7252dfa1813 with delete_type TABLET_DATA_DELETED (replaced by tablet b3c5a861a90840d0a33b780ec75bffd2) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.962309 746 tablet_service.cc:1514] Processing DeleteTablet for tablet b7bd96fb381f4e73a0081be3fc799c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.962445 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 42cb7e7a41634cf1bdb978ac9d2befcd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.962579 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f59617b34046568dd83bbb7cdc42d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.962707 743 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a3486ef011492c8a1df8d6278adedf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.964502 747 tablet_service.cc:1514] Processing DeleteTablet for tablet a12c5d58e6c246c983e8a533906ce77d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.965735 743 tablet_service.cc:1514] Processing DeleteTablet for tablet fb0aba0139cf479b80b32f3dfc936535 with delete_type TABLET_DATA_DELETED (replaced by tablet e2f1f1a76e5f4308bf56dd647f02c6d8) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.965932 756 tablet_service.cc:1514] Processing DeleteTablet for tablet e5c10fad790046a88fbc8ebe0303c481 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.966094 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet cb724feccc7f4fbc846fb6fa61be3385 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.967656 746 tablet_service.cc:1514] Processing DeleteTablet for tablet ea33e5c1a1fb46599c5836a36b130051 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.967919 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 015412a5fa304453bd8d1be1c3cd7a46 with delete_type TABLET_DATA_DELETED (replaced by tablet f924459d1150472aafb325a11825db23) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.968168 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 64707ee432fb4e96a3bbf7252dfa1813 with delete_type TABLET_DATA_DELETED (replaced by tablet b3c5a861a90840d0a33b780ec75bffd2) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.968770 1173 ts_tablet_manager.cc:1428] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250112 02:52:38.968894 1181 tablet_bootstrap.cc:492] T d63d411f906b4d4eb68343bdcc493e05 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:38.969017 1181 ts_tablet_manager.cc:1397] T d63d411f906b4d4eb68343bdcc493e05 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.073s user 0.002s sys 0.003s
I20250112 02:52:38.969259 899 tablet_service.cc:1514] Processing DeleteTablet for tablet e5c10fad790046a88fbc8ebe0303c481 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.969564 899 tablet_service.cc:1514] Processing DeleteTablet for tablet d4abf59d96414e5faa7cf75ba133832c with delete_type TABLET_DATA_DELETED (replaced by tablet 5a3af0ea4e954053ae89814f3e74e6b7) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.969640 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a12b225131b464494a9f02133d37aea with delete_type TABLET_DATA_DELETED (replaced by tablet d45fce86e02f46d4a57af718fa2a62c8) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.969816 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd48d0e68014e1aa0c44620527baec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.969785 1181 raft_consensus.cc:357] T d63d411f906b4d4eb68343bdcc493e05 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.969941 1181 raft_consensus.cc:383] T d63d411f906b4d4eb68343bdcc493e05 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:38.969966 898 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a3486ef011492c8a1df8d6278adedf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.969993 1181 raft_consensus.cc:738] T d63d411f906b4d4eb68343bdcc493e05 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:38.970129 889 tablet_service.cc:1514] Processing DeleteTablet for tablet ac716c6da01941a989754aac1ef430d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.970281 887 tablet_service.cc:1514] Processing DeleteTablet for tablet f05c8426ea3d4fe89fbe6c88c688019f with delete_type TABLET_DATA_DELETED (replaced by tablet c5079f87652a4ea58b366d1f9448fd4e) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.970696 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d662e0ce13f447595e5f92b94de3018" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:38.970486 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 416fb6820cd54e118950c68d381f51b2) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.971024 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet c7c530aa0aa64e47ae27d61f341d2bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.971189 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1e3af2883341e9b957f6ed748e604d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.971215 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 1726d61cc36a4deb83c5d604e8daf4f0 with delete_type TABLET_DATA_DELETED (replaced by tablet b2ccb88108f24e36ba4b7ebf36398950) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.971379 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet f7cd0927c6a4401cbe69f55d4db9963a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.971840 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd48d0e68014e1aa0c44620527baec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.972018 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d5b3df88a0c45dba879b3d506710614 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.972158 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b599e983ec04f53a440d1ab828d12ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.972385 746 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d84027ad6d40a5be5afcdd854431a5 with delete_type TABLET_DATA_DELETED (replaced by tablet fbe9e9624ede40c29ea852b2b5494965) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.972491 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 9163878cfeb64693a9c5493e812546af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.972527 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet b7bd96fb381f4e73a0081be3fc799c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.973083 889 tablet_service.cc:1514] Processing DeleteTablet for tablet ea33e5c1a1fb46599c5836a36b130051 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.973279 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 416fb6820cd54e118950c68d381f51b2) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.973358 760 tablet_service.cc:1514] Processing DeleteTablet for tablet a85300a9aa1c4c50a3957a004bea7933 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.975749 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 352c87f1dc724e7b94bc2e64abfce8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.975901 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a34d03a80ae402db6607261a5c40b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.976254 1173 tablet_bootstrap.cc:492] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:38.977236 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003a3b4705e943c6a900762ec11ddd47" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:38.977464 1049 raft_consensus.cc:2463] T 003a3b4705e943c6a900762ec11ddd47 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
W20250112 02:52:38.978112 855 leader_election.cc:343] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.978200 855 leader_election.cc:304] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:38.978379 2399 raft_consensus.cc:2743] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:38.978451 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795ac7eaf81d48bb926865d9ebb388b6" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:38.978653 855 leader_election.cc:343] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:38.978739 855 leader_election.cc:304] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:38.978931 2399 raft_consensus.cc:2743] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:38.979230 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dda52f012794493a102ed1abc718f5b with delete_type TABLET_DATA_DELETED (replaced by tablet 17c54c5dc86446cf85974777c8145ef6) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.981848 887 tablet_service.cc:1514] Processing DeleteTablet for tablet c04b64bc2f8348a086ed3c4e59f09b16 with delete_type TABLET_DATA_DELETED (replaced by tablet 61eebc5f22664712af6e74e82809ca87) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.982820 887 tablet_service.cc:1514] Processing DeleteTablet for tablet bb217afc08304c0ea35dee3120347ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.983572 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b599e983ec04f53a440d1ab828d12ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.984371 887 tablet_service.cc:1514] Processing DeleteTablet for tablet a85300a9aa1c4c50a3957a004bea7933 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:38.942319 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 74) took 4421 ms. Trace:
I20250112 02:52:38.984635 2024 rpcz_store.cc:276] 0112 02:52:34.520287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.520345 (+ 58us) service_pool.cc:225] Handling call
0112 02:52:38.867721 (+4347376us) ts_tablet_manager.cc:1145] Deleting tablet 9d5b3df88a0c45dba879b3d506710614
0112 02:52:38.867733 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:38.881794 (+ 14061us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.909148 (+ 27354us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:38.941661 (+ 32513us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4325131}
I20250112 02:52:38.984863 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 54215c55966a46e1ab96e352c4e3764e with delete_type TABLET_DATA_DELETED (replaced by tablet a470f84d50df4cf0b4a1e5d248b4c32a) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.985011 2024 tablet_replica.cc:331] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:38.985348 2024 raft_consensus.cc:2238] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.985585 2024 raft_consensus.cc:2267] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.986158 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 04f135b226774285a077e5806131d91b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.970182 1181 consensus_queue.cc:260] T d63d411f906b4d4eb68343bdcc493e05 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:38.986265 2024 ts_tablet_manager.cc:1905] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.981772 714 leader_election.cc:304] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:38.986826 1181 ts_tablet_manager.cc:1428] T d63d411f906b4d4eb68343bdcc493e05 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.018s user 0.000s sys 0.001s
I20250112 02:52:38.986848 2047 raft_consensus.cc:2798] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:38.986912 2047 raft_consensus.cc:491] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:38.986958 2047 raft_consensus.cc:3054] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.987052 1181 tablet_bootstrap.cc:492] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:38.987603 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b184806b4884cd3b0a002905a1e57af with delete_type TABLET_DATA_DELETED (replaced by tablet 47a0d939fdb94590a86389f1f4693e9a) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:38.989001 2562 ts_tablet_manager.cc:1918] T bd5b3c712c0f448ea9b14ab93dd731ab P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:38.989085 2562 log.cc:1198] T bd5b3c712c0f448ea9b14ab93dd731ab P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/bd5b3c712c0f448ea9b14ab93dd731ab
I20250112 02:52:38.989346 2562 ts_tablet_manager.cc:1939] T bd5b3c712c0f448ea9b14ab93dd731ab P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:38.989945 714 leader_election.cc:304] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
W20250112 02:52:38.977058 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 8c821ea3ca4946c290c17135366e0cac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c821ea3ca4946c290c17135366e0cac already in progress: creating tablet
I20250112 02:52:38.990944 1181 tablet_bootstrap.cc:654] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:38.985260 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 9d5b3df88a0c45dba879b3d506710614 (table default.Table_49b5979e0b174043b61f2f1d61a5b58c [id=b81cf271b11e45dcb0b6bce102cd7a0c]) successfully deleted
I20250112 02:52:38.991658 2003 raft_consensus.cc:2238] T 8ee98f055024429eafaa29bb961cab87 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:38.991825 2003 raft_consensus.cc:2267] T 8ee98f055024429eafaa29bb961cab87 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:38.992012 2570 raft_consensus.cc:3054] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:38.992187 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet aadb2ab1d2024f889841ac91e065e064 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.992530 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a0a13696fc41ccb28218f1275a4b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.992969 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 41e166b3f8414ababbd4a3f07fb90c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.993320 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet a47cdc208fbc431b8326364e84432cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.993470 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet fe614eaeb793473d9b2da206b7345562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.993629 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0c2fad4f3243238641aae56b305bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:38.992347 2003 ts_tablet_manager.cc:1905] T 8ee98f055024429eafaa29bb961cab87 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:38.999140 1173 tablet_bootstrap.cc:654] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.000465 1181 tablet_bootstrap.cc:492] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:39.000577 1181 ts_tablet_manager.cc:1397] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s
I20250112 02:52:39.001305 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f59617b34046568dd83bbb7cdc42d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.001273 1181 raft_consensus.cc:357] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:39.001390 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet b2d8ea2ef18a46508371b4789ab6f4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.001425 1181 raft_consensus.cc:383] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.001474 1181 raft_consensus.cc:738] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.001490 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 42cb7e7a41634cf1bdb978ac9d2befcd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.001672 1181 consensus_queue.cc:260] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:39.002250 1181 ts_tablet_manager.cc:1428] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:39.003012 1181 tablet_bootstrap.cc:492] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:39.003697 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af55dab202444358ea1a0674d37946d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:39.004388 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet ac716c6da01941a989754aac1ef430d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.004568 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet f6803d0f553b4204b415b2734286bf4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.004669 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet cc62dcbf67e84e36ae1fbe127ccd4879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.004956 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c686fdc8896463c81c9f7c5f53a3da2 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c03b5e612d64314b210c11686fdfa2a) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.005038 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b521522c4784d2ab4056268823f1099 with delete_type TABLET_DATA_DELETED (replaced by tablet ab2e31a65294407bb62b13e92898a7d2) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.005228 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f4e4f1e451b48dc9cdd780300f6888c with delete_type TABLET_DATA_DELETED (replaced by tablet 2052b4536f9745f0a785f330d706c1cf) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.005398 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a3486ef011492c8a1df8d6278adedf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.005640 1181 tablet_bootstrap.cc:654] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.010128 2562 tablet_replica.cc:331] T 7e1954e7859f422f9690fe4a9384a8bb P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:39.015081 2024 ts_tablet_manager.cc:1918] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:39.015393 2562 raft_consensus.cc:2238] T 7e1954e7859f422f9690fe4a9384a8bb P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:39.015808 2562 raft_consensus.cc:2267] T 7e1954e7859f422f9690fe4a9384a8bb P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.017095 2562 ts_tablet_manager.cc:1905] T 7e1954e7859f422f9690fe4a9384a8bb P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.017987 2047 raft_consensus.cc:513] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.017520 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 73136be25c79491784d3ea71aa9b27f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:39.020763 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2f1f1a76e5f4308bf56dd647f02c6d8" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:39.020977 771 raft_consensus.cc:3054] T e2f1f1a76e5f4308bf56dd647f02c6d8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.023952 771 raft_consensus.cc:2463] T e2f1f1a76e5f4308bf56dd647f02c6d8 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:39.024217 747 tablet_service.cc:1514] Processing DeleteTablet for tablet d4abf59d96414e5faa7cf75ba133832c with delete_type TABLET_DATA_DELETED (replaced by tablet 5a3af0ea4e954053ae89814f3e74e6b7) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:39.024245 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet bd5b3c712c0f448ea9b14ab93dd731ab (table default.Table_9c10574f7c7d435890eb9a24af23bcb9 [id=c0e16b77525146a28b8182137bc81566]) successfully deleted
I20250112 02:52:39.024405 744 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a7281c0c09442ebd0cc13b8496c86e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:39.024652 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb76746b3a846c3aba531c04bf63966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:39.024269 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b1d9fae290ec4281839991f9d40b105c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1d9fae290ec4281839991f9d40b105c already in progress: creating tablet
I20250112 02:52:39.025440 891 tablet_service.cc:1514] Processing DeleteTablet for tablet a12c5d58e6c246c983e8a533906ce77d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
W20250112 02:52:39.025504 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b already in progress: creating tablet
I20250112 02:52:39.025625 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b8daf4ef3593410eae87672402292be6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:39.025813 891 tablet_service.cc:1514] Processing DeleteTablet for tablet f234256b343f447dadfc3d1936a824ee with delete_type TABLET_DATA_DELETED (replaced by tablet 82b351c563d749df806ab5d0331a4ef2) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:39.025853 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d5b3df88a0c45dba879b3d506710614 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:39.026038 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b184806b4884cd3b0a002905a1e57af with delete_type TABLET_DATA_DELETED (replaced by tablet 47a0d939fdb94590a86389f1f4693e9a) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:39.026182 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 352c87f1dc724e7b94bc2e64abfce8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:39.026299 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 73136be25c79491784d3ea71aa9b27f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:39.026404 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb76746b3a846c3aba531c04bf63966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:39.026549 899 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a7281c0c09442ebd0cc13b8496c86e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
W20250112 02:52:39.029002 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f776e2a6ec704cf9b64483bc7d5b1dff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f776e2a6ec704cf9b64483bc7d5b1dff already in progress: creating tablet
I20250112 02:52:39.029079 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795ac7eaf81d48bb926865d9ebb388b6" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:39.029211 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 62f4bd2df0ce45dabc55373f1154d3ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62f4bd2df0ce45dabc55373f1154d3ac already in progress: creating tablet
I20250112 02:52:39.030089 2179 consensus_queue.cc:1035] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:52:39.030230 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 058a9ca089cd4967887b0a1127c7d03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 058a9ca089cd4967887b0a1127c7d03f already in progress: creating tablet
W20250112 02:52:39.030421 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet a8e7882e07784bdda18dc1ffd20963c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8e7882e07784bdda18dc1ffd20963c8 already in progress: creating tablet
I20250112 02:52:39.031028 2047 leader_election.cc:290] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.031441 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a18132549374218893f35a4e4226b72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:39.031603 713 leader_election.cc:343] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 003a3b4705e943c6a900762ec11ddd47
I20250112 02:52:39.031682 747 tablet_service.cc:1514] Processing DeleteTablet for tablet b8daf4ef3593410eae87672402292be6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:39.031637 2570 raft_consensus.cc:2743] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:39.031773 451 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 003a3b4705e943c6a900762ec11ddd47 (table default.Table_743b3f7029534564b35c0c137b35fd75 [id=b2fa3922f6d447769a8179052f9a1861], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 003a3b4705e943c6a900762ec11ddd47 consensus state has no leader
I20250112 02:52:39.031908 746 tablet_service.cc:1514] Processing DeleteTablet for tablet d45fce86e02f46d4a57af718fa2a62c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 5804fc4130f84c69832c1022b7bd7fa5) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:39.032338 775 raft_consensus.cc:2463] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:39.032994 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c821ea3ca4946c290c17135366e0cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:39.033192 746 tablet_service.cc:1514] Processing DeleteTablet for tablet f234256b343f447dadfc3d1936a824ee with delete_type TABLET_DATA_DELETED (replaced by tablet 82b351c563d749df806ab5d0331a4ef2) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:39.033718 744 tablet_service.cc:1514] Processing DeleteTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:39.033895 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:39.034042 746 tablet_service.cc:1514] Processing DeleteTablet for tablet b1d9fae290ec4281839991f9d40b105c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:39.034627 1177 tablet_bootstrap.cc:492] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:39.034734 1177 ts_tablet_manager.cc:1397] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.277s user 0.022s sys 0.029s
I20250112 02:52:39.035416 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4ef0220dde43569452379639759db5" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:39.038156 2570 consensus_queue.cc:1035] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:39.043820 1177 raft_consensus.cc:357] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.044008 1177 raft_consensus.cc:383] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.044062 1177 raft_consensus.cc:738] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.044260 1177 consensus_queue.cc:260] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.044827 1177 ts_tablet_manager.cc:1428] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20250112 02:52:39.045112 1177 tablet_bootstrap.cc:492] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:39.029580 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795ac7eaf81d48bb926865d9ebb388b6" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:39.048755 771 raft_consensus.cc:2463] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:39.053293 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4ef0220dde43569452379639759db5" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
W20250112 02:52:39.053329 452 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:37987 (request call id 253) took 2245 ms (client timeout 1000 ms). Trace:
I20250112 02:52:39.054467 2570 consensus_queue.cc:1035] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:39.066095 1177 tablet_bootstrap.cc:654] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.075057 2399 raft_consensus.cc:491] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.075177 2399 raft_consensus.cc:513] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:39.075748 2399 leader_election.cc:290] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:39.078083 451 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:37987 (request call id 251) took 5021 ms (client timeout 1000 ms). Trace:
W20250112 02:52:39.078248 451 rpcz_store.cc:269] 0112 02:52:34.056137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.056285 (+ 148us) service_pool.cc:225] Handling call
0112 02:52:39.065740 (+5009455us) spinlock_profiling.cc:243] Waited 4.97 ms on lock 0x60f00002f058. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 000000000096ceda 00007f3e098520fd 00007f3e09851d1e 00007f3e1cc38f3d 00007f3e1cc38edc 00007f3e0aa2e813 00007f3e1cd700eb 00007f3e13e72716 00007f3e13e72445 00007f3e105d130e 00007f3e105d01ee
0112 02:52:39.078051 (+ 12311us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":122596,"reported_tablets":172,"spinlock_wait_cycles":11768320}
I20250112 02:52:39.079041 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a3af0ea4e954053ae89814f3e74e6b7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:39.079905 855 leader_election.cc:343] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:39.080873 713 consensus_peers.cc:487] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5c10fad790046a88fbc8ebe0303c481. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:52:39.085937 453 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:58959 (request call id 237) took 4582 ms (client timeout 1000 ms). Trace:
W20250112 02:52:39.086097 453 rpcz_store.cc:269] 0112 02:52:34.485626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.555569 (+ 69943us) service_pool.cc:225] Handling call
0112 02:52:39.067647 (+4512078us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:36.461640 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:36.461803 (+ 163us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:36.461833 (+ 30us) write_op.cc:435] Acquired schema lock
0112 02:52:36.461838 (+ 5us) tablet.cc:659] Decoding operations
0112 02:52:36.461995 (+ 157us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:36.462028 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:36.462033 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:36.462106 (+ 73us) tablet.cc:698] Row locks acquired
0112 02:52:36.462108 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:36.462154 (+ 46us) write_op.cc:270] Start()
0112 02:52:36.462191 (+ 37us) write_op.cc:276] Timestamp: P: 1736650356462035 usec, L: 0
0112 02:52:36.462195 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:36.669015 (+206820us) log.cc:844] Serialized 356 byte log entry
0112 02:52:36.669167 (+ 152us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 000000000096ceda 00007f3e150e3632 00007f3e1646beb5 00007f3e1646a9d9 00007f3e16472e00 00007f3e16472c0d 0000000000a71ecc 00007f3e098b0b64 00007f3e098b418b 00007f3e098b4008 0000000000a71ecc
0112 02:52:36.680277 (+ 11110us) op_driver.cc:464] REPLICATION: finished
0112 02:52:36.680344 (+ 67us) write_op.cc:301] APPLY: starting
0112 02:52:36.680356 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:36.680386 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:36.680389 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:36.680436 (+ 47us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:36.680490 (+ 54us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:36.680499 (+ 9us) write_op.cc:312] APPLY: finished
0112 02:52:36.680554 (+ 55us) log.cc:844] Serialized 23 byte log entry
0112 02:52:36.680601 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.680615 (+ 14us) write_op.cc:454] Released schema lock
0112 02:52:36.680621 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":80483,"reported_tablets":171,"child_traces":[["op",{"apply.queue_time_us":29,"apply.run_cpu_time_us":331,"apply.run_wall_time_us":330,"num_ops":1,"prepare.queue_time_us":621154,"prepare.run_cpu_time_us":2481,"prepare.run_wall_time_us":207619,"raft.queue_time_us":1168,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":290,"replication_time_us":218057,"spinlock_wait_cycles":475219328}]]}
I20250112 02:52:39.087981 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_32f2fd919490410aacdc9ca9c5a51683" } new_table_name: "default.Table_8fe6c818f15341e08fe8870664eac4c1" modify_external_catalogs: true
I20250112 02:52:39.090124 1181 tablet_bootstrap.cc:492] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:39.090245 1181 ts_tablet_manager.cc:1397] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.087s user 0.014s sys 0.014s
I20250112 02:52:39.090885 1181 raft_consensus.cc:357] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.091049 1181 raft_consensus.cc:383] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.091105 1181 raft_consensus.cc:738] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.091259 1181 consensus_queue.cc:260] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.091859 1181 ts_tablet_manager.cc:1428] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:39.094931 1181 tablet_bootstrap.cc:492] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:39.097716 1181 tablet_bootstrap.cc:654] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.099332 2570 consensus_queue.cc:1035] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20250112 02:52:39.099740 2179 consensus_queue.cc:1035] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:52:39.103367 446 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:58959 (request call id 238) took 2990 ms (client timeout 1000 ms). Trace:
W20250112 02:52:39.103518 446 rpcz_store.cc:269] 0112 02:52:36.069652 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:36.416720 (+347068us) service_pool.cc:225] Handling call
0112 02:52:36.428702 (+ 11982us) spinlock_profiling.cc:243] Waited 7.76 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd6d6b8 00007f3e13e72716 00007f3e13e72445 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:38.943144 (+2514442us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x6130001737a0. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 000000000096ceda 00007f3e1053c4cd 00007f3e13205569 00007f3e1cc6736f 00007f3e1cbb5725 00007f3e1cbcb4fc 00007f3e1cd6f847 00007f3e13e72716 00007f3e13e72445 00007f3e105d130e 00007f3e105d01ee
0112 02:52:39.060461 (+117317us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:37.348666 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:37.348770 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:37.348776 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:37.348779 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:37.348883 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:37.348896 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:37.348900 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:37.348958 (+ 58us) tablet.cc:698] Row locks acquired
0112 02:52:37.348960 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:37.348996 (+ 36us) write_op.cc:270] Start()
0112 02:52:37.349027 (+ 31us) write_op.cc:276] Timestamp: P: 1736650357348924 usec, L: 0
0112 02:52:37.349048 (+ 21us) op_driver.cc:348] REPLICATION: starting
0112 02:52:37.349216 (+ 168us) log.cc:844] Serialized 356 byte log entry
0112 02:52:37.354209 (+ 4993us) op_driver.cc:464] REPLICATION: finished
0112 02:52:37.354348 (+ 139us) write_op.cc:301] APPLY: starting
0112 02:52:37.354364 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:37.354436 (+ 72us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:37.354478 (+ 42us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:37.354557 (+ 79us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:37.354601 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:37.354612 (+ 11us) write_op.cc:312] APPLY: finished
0112 02:52:37.354693 (+ 81us) log.cc:844] Serialized 23 byte log entry
0112 02:52:37.354760 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:37.354781 (+ 21us) write_op.cc:454] Released schema lock
0112 02:52:37.354791 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:37.354876 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:37.354882 (+ 6us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:38.241211 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:38.241313 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:38.241319 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:38.241322 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:38.241430 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:38.241445 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:38.241449 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:38.241506 (+ 57us) tablet.cc:698] Row locks acquired
0112 02:52:38.241508 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:38.241543 (+ 35us) write_op.cc:270] Start()
0112 02:52:38.241572 (+ 29us) write_op.cc:276] Timestamp: P: 1736650358241499 usec, L: 0
0112 02:52:38.241576 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:38.241750 (+ 174us) log.cc:844] Serialized 356 byte log entry
0112 02:52:38.244959 (+ 3209us) op_driver.cc:464] REPLICATION: finished
0112 02:52:38.245050 (+ 91us) write_op.cc:301] APPLY: starting
0112 02:52:38.245061 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:38.245098 (+ 37us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:38.245119 (+ 21us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:38.245155 (+ 36us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:38.245183 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:38.245189 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:38.245240 (+ 51us) log.cc:844] Serialized 23 byte log entry
0112 02:52:38.245284 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.245295 (+ 11us) write_op.cc:454] Released schema lock
0112 02:52:38.245299 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:38.245361 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.245364 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":23112,"reported_tablets":198,"spinlock_wait_cycles":41196544,"child_traces":[["op",{"apply.queue_time_us":86,"apply.run_cpu_time_us":513,"apply.run_wall_time_us":514,"num_ops":1,"prepare.queue_time_us":8012,"prepare.run_cpu_time_us":655,"prepare.run_wall_time_us":655,"raft.queue_time_us":365,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":193,"replication_time_us":5126}],["op",{"apply.queue_time_us":53,"apply.run_cpu_time_us":303,"apply.run_wall_time_us":303,"num_ops":1,"prepare.queue_time_us":11321,"prepare.run_cpu_time_us":645,"prepare.run_wall_time_us":645,"raft.queue_time_us":839,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":206,"replication_time_us":3359}]]}
I20250112 02:52:39.109815 2496 raft_consensus.cc:491] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.109941 2496 raft_consensus.cc:513] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.110466 2496 leader_election.cc:290] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:39.110956 2496 raft_consensus.cc:491] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.111050 2496 raft_consensus.cc:513] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.111645 2496 leader_election.cc:290] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:39.113745 2003 ts_tablet_manager.cc:1918] T 8ee98f055024429eafaa29bb961cab87 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250112 02:52:39.053427 452 rpcz_store.cc:269] 0112 02:52:36.807658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:36.820771 (+ 13113us) service_pool.cc:225] Handling call
0112 02:52:37.820134 (+999363us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:37.820139 (+ 5us) write_op.cc:454] Released schema lock
0112 02:52:39.052746 (+1232607us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:37.814162 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:37.814274 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:37.814283 (+ 9us) write_op.cc:435] Acquired schema lock
0112 02:52:37.814287 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:37.814400 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:37.814413 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:37.814418 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:37.814522 (+ 104us) tablet.cc:698] Row locks acquired
0112 02:52:37.814524 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:37.814572 (+ 48us) write_op.cc:270] Start()
0112 02:52:37.814618 (+ 46us) write_op.cc:276] Timestamp: P: 1736650357814452 usec, L: 0
0112 02:52:37.814621 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:37.814781 (+ 160us) log.cc:844] Serialized 901 byte log entry
0112 02:52:37.817923 (+ 3142us) op_driver.cc:464] REPLICATION: finished
0112 02:52:37.818017 (+ 94us) write_op.cc:301] APPLY: starting
0112 02:52:37.818031 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:37.818070 (+ 39us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:37.818099 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:37.818158 (+ 59us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:37.818187 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:37.818192 (+ 5us) write_op.cc:312] APPLY: finished
0112 02:52:37.818252 (+ 60us) log.cc:844] Serialized 29 byte log entry
0112 02:52:37.818303 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:37.818320 (+ 17us) write_op.cc:454] Released schema lock
0112 02:52:37.818326 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 02:52:38.253749 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:38.253829 (+ 80us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:38.253834 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:38.253837 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:38.253910 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:38.253920 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:38.253926 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:38.253984 (+ 58us) tablet.cc:698] Row locks acquired
0112 02:52:38.253986 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:38.254010 (+ 24us) write_op.cc:270] Start()
0112 02:52:38.254033 (+ 23us) write_op.cc:276] Timestamp: P: 1736650358253966 usec, L: 0
0112 02:52:38.254036 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:38.254162 (+ 126us) log.cc:844] Serialized 356 byte log entry
0112 02:52:38.257392 (+ 3230us) op_driver.cc:464] REPLICATION: finished
0112 02:52:38.257498 (+ 106us) write_op.cc:301] APPLY: starting
0112 02:52:38.257510 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:38.257533 (+ 23us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:38.257558 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:38.257591 (+ 33us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:38.257611 (+ 20us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:38.257616 (+ 5us) write_op.cc:312] APPLY: finished
0112 02:52:38.257668 (+ 52us) log.cc:844] Serialized 23 byte log entry
0112 02:52:38.257711 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.257722 (+ 11us) write_op.cc:454] Released schema lock
0112 02:52:38.257727 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:38.257781 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.257786 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:38.260146 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:38.260213 (+ 67us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:38.260218 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:38.260219 (+ 1us) tablet.cc:659] Decoding operations
0112 02:52:38.260295 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:38.260303 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:38.260306 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:38.260342 (+ 36us) tablet.cc:698] Row locks acquired
0112 02:52:38.260344 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:38.260365 (+ 21us) write_op.cc:270] Start()
0112 02:52:38.260384 (+ 19us) write_op.cc:276] Timestamp: P: 1736650358260321 usec, L: 0
0112 02:52:38.260387 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:38.260492 (+ 105us) log.cc:844] Serialized 356 byte log entry
0112 02:52:38.274574 (+ 14082us) op_driver.cc:464] REPLICATION: finished
0112 02:52:38.274702 (+ 128us) write_op.cc:301] APPLY: starting
0112 02:52:38.274726 (+ 24us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:38.274754 (+ 28us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:38.274777 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:38.274808 (+ 31us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:38.274832 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:38.274840 (+ 8us) write_op.cc:312] APPLY: finished
0112 02:52:38.274884 (+ 44us) log.cc:844] Serialized 23 byte log entry
0112 02:52:38.274935 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.274946 (+ 11us) write_op.cc:454] Released schema lock
0112 02:52:38.274951 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:38.275006 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.275009 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:38.278678 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:38.278760 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:38.278766 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:38.278770 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:38.278835 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:38.278843 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:38.278847 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:38.278887 (+ 40us) tablet.cc:698] Row locks acquired
0112 02:52:38.278890 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:38.278911 (+ 21us) write_op.cc:270] Start()
0112 02:52:38.278928 (+ 17us) write_op.cc:276] Timestamp: P: 1736650358278863 usec, L: 0
0112 02:52:38.278933 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:38.279063 (+ 130us) log.cc:844] Serialized 356 byte log entry
0112 02:52:38.282278 (+ 3215us) op_driver.cc:464] REPLICATION: finished
0112 02:52:38.282374 (+ 96us) write_op.cc:301] APPLY: starting
0112 02:52:38.282385 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:38.282414 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:38.282446 (+ 32us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:38.282487 (+ 41us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:38.282510 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:38.282528 (+ 18us) write_op.cc:312] APPLY: finished
0112 02:52:38.282597 (+ 69us) log.cc:844] Serialized 23 byte log entry
0112 02:52:38.282653 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.282686 (+ 33us) write_op.cc:454] Released schema lock
0112 02:52:38.282703 (+ 17us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:38.282753 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.282757 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:38.676537 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:38.676656 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:38.676664 (+ 8us) write_op.cc:435] Acquired schema lock
0112 02:52:38.676668 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:38.676804 (+ 136us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:38.676819 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:38.676825 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:38.676886 (+ 61us) tablet.cc:698] Row locks acquired
0112 02:52:38.676889 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:38.676925 (+ 36us) write_op.cc:270] Start()
0112 02:52:38.676960 (+ 35us) write_op.cc:276] Timestamp: P: 1736650358676893 usec, L: 0
0112 02:52:38.676964 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:38.677168 (+ 204us) log.cc:844] Serialized 356 byte log entry
0112 02:52:38.680705 (+ 3537us) op_driver.cc:464] REPLICATION: finished
0112 02:52:38.680849 (+ 144us) write_op.cc:301] APPLY: starting
0112 02:52:38.680866 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:38.680911 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:38.680944 (+ 33us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:38.680998 (+ 54us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:38.681030 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:38.681037 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:38.681106 (+ 69us) log.cc:844] Serialized 23 byte log entry
0112 02:52:38.681164 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.681182 (+ 18us) write_op.cc:454] Released schema lock
0112 02:52:38.681189 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:38.681260 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.681265 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:38.683991 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:38.684109 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:38.684114 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:38.684116 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:38.684197 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:38.684209 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:38.684216 (+ 7us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:38.684267 (+ 51us) tablet.cc:698] Row locks acquired
0112 02:52:38.684269 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:38.684296 (+ 27us) write_op.cc:270] Start()
0112 02:52:38.684320 (+ 24us) write_op.cc:276] Timestamp: P: 1736650358684264 usec, L: 0
0112 02:52:38.684324 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:38.684457 (+ 133us) log.cc:844] Serialized 356 byte log entry
0112 02:52:38.695027 (+ 10570us) op_driver.cc:464] REPLICATION: finished
0112 02:52:38.695135 (+ 108us) write_op.cc:301] APPLY: starting
0112 02:52:38.695148 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:38.695184 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:38.695209 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:38.695248 (+ 39us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:38.695277 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:38.695286 (+ 9us) write_op.cc:312] APPLY: finished
0112 02:52:38.695357 (+ 71us) log.cc:844] Serialized 23 byte log entry
0112 02:52:38.695410 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.695425 (+ 15us) write_op.cc:454] Released schema lock
0112 02:52:38.695430 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:38.695513 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.695519 (+ 6us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:38.698527 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:38.698610 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:38.698615 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:38.698617 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:38.698684 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:38.698692 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:38.698696 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:38.698745 (+ 49us) tablet.cc:698] Row locks acquired
0112 02:52:38.698747 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:38.698769 (+ 22us) write_op.cc:270] Start()
0112 02:52:38.698790 (+ 21us) write_op.cc:276] Timestamp: P: 1736650358698738 usec, L: 0
0112 02:52:38.698794 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:38.698908 (+ 114us) log.cc:844] Serialized 356 byte log entry
0112 02:52:38.704572 (+ 5664us) op_driver.cc:464] REPLICATION: finished
0112 02:52:38.704673 (+ 101us) write_op.cc:301] APPLY: starting
0112 02:52:38.704687 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:38.704729 (+ 42us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:38.704761 (+ 32us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:38.704802 (+ 41us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:38.704828 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:38.704836 (+ 8us) write_op.cc:312] APPLY: finished
0112 02:52:38.704894 (+ 58us) log.cc:844] Serialized 23 byte log entry
0112 02:52:38.704947 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.704962 (+ 15us) write_op.cc:454] Released schema lock
0112 02:52:38.704968 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:38.705027 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.705032 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:38.713273 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:38.713409 (+ 136us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:38.713415 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:38.713419 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:38.713529 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:38.713542 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:38.713549 (+ 7us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:38.713605 (+ 56us) tablet.cc:698] Row locks acquired
0112 02:52:38.713609 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:38.713654 (+ 45us) write_op.cc:270] Start()
0112 02:52:38.713681 (+ 27us) write_op.cc:276] Timestamp: P: 1736650358713603 usec, L: 0
0112 02:52:38.713684 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:38.713845 (+ 161us) log.cc:844] Serialized 356 byte log entry
0112 02:52:38.716030 (+ 2185us) op_driver.cc:464] REPLICATION: finished
0112 02:52:38.716143 (+ 113us) write_op.cc:301] APPLY: starting
0112 02:52:38.716157 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:38.716191 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:38.716221 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:38.716269 (+ 48us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:38.716293 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:38.716300 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:38.716357 (+ 57us) log.cc:844] Serialized 23 byte log entry
0112 02:52:38.716410 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.716427 (+ 17us) write_op.cc:454] Released schema lock
0112 02:52:38.716434 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:38.716503 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.716509 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7637,"reported_tablets":210,"child_traces":[["op",{"apply.queue_time_us":57,"apply.run_cpu_time_us":364,"apply.run_wall_time_us":363,"num_ops":2,"prepare.queue_time_us":920,"prepare.run_cpu_time_us":708,"prepare.run_wall_time_us":709,"raft.queue_time_us":358,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":192,"replication_time_us":3267}],["op",{"apply.queue_time_us":59,"apply.run_cpu_time_us":288,"apply.run_wall_time_us":288,"num_ops":1,"prepare.queue_time_us":37,"prepare.run_cpu_time_us":483,"prepare.run_wall_time_us":483,"raft.queue_time_us":47,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":149,"replication_time_us":3334}],["op",{"apply.queue_time_us":84,"apply.run_cpu_time_us":301,"apply.run_wall_time_us":301,"num_ops":1,"prepare.queue_time_us":149,"prepare.run_cpu_time_us":414,"prepare.run_wall_time_us":414,"raft.queue_time_us":49,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":156,"replication_time_us":14164}],["op",{"apply.queue_time_us":60,"apply.run_cpu_time_us":372,"apply.run_wall_time_us":372,"num_ops":1,"prepare.queue_time_us":966,"prepare.run_cpu_time_us":462,"prepare.run_wall_time_us":462,"raft.queue_time_us":53,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":160,"replication_time_us":3319}],["op",{"apply.queue_time_us":73,"apply.run_cpu_time_us":401,"apply.run_wall_time_us":400,"num_ops":1,"prepare.queue_time_us":2416,"prepare.run_cpu_time_us":742,"prepare.run_wall_time_us":742,"raft.queue_time_us":335,"raft.run_cpu_time_us":348,"raft.run_wall_time_us":349,"replication_time_us":3715,"spinlock_wait_cycles":185984}],["op",{"apply.queue_time_us":60,"apply.run_cpu_time_us":369,"apply.run_wall_time_us":368,"num_ops":1,"prepare.queue_time_us":36,"prepare.run_cpu_time_us":554,"prepare.run_wall_time_us":553,"raft.queue_time_us":57,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":203,"replication_time_us":10679}],["op",{"apply.queue_time_us":61,"apply.run_cpu_time_us":347,"apply.run_wall_time_us":347,"num_ops":1,"prepare.queue_time_us":90,"prepare.run_cpu_time_us":458,"prepare.run_wall_time_us":458,"raft.queue_time_us":47,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":166,"replication_time_us":5748}],["op",{"apply.queue_time_us":64,"apply.run_cpu_time_us":352,"apply.run_wall_time_us":351,"num_ops":1,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":666,"prepare.run_wall_time_us":666,"raft.queue_time_us":68,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":219,"replication_time_us":2317}]]}
I20250112 02:52:39.115762 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f776e2a6ec704cf9b64483bc7d5b1dff" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:39.112380 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d5799b5b36487da42dc43b1c70ae26" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:39.118533 855 leader_election.cc:343] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:39.118796 855 leader_election.cc:343] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.119161 827 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:39.120931 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_48ca697307d74d89aa61ef5432217ca9" } modify_external_catalogs: true
I20250112 02:52:39.121197 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_48ca697307d74d89aa61ef5432217ca9" } modify_external_catalogs: true
I20250112 02:52:39.119617 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_0bdc5f49d4ff4d6fb1290bc2dc0f70d9" } new_table_name: "default.Table_2c52fead633649dd869763e5aa532be7" modify_external_catalogs: true
I20250112 02:52:39.123404 1181 tablet_bootstrap.cc:492] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:39.123544 1181 ts_tablet_manager.cc:1397] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.006s
I20250112 02:52:39.124253 1181 raft_consensus.cc:357] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.124421 1181 raft_consensus.cc:383] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.124496 1181 raft_consensus.cc:738] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.124701 1181 consensus_queue.cc:260] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.125259 1181 ts_tablet_manager.cc:1428] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:39.125486 1181 tablet_bootstrap.cc:492] T 709517ae520640208abe85fa14a5301e P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:39.126994 1181 tablet_bootstrap.cc:654] T 709517ae520640208abe85fa14a5301e P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:39.134102 447 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:38361 (request call id 211) took 2741 ms (client timeout 1000 ms). Trace:
W20250112 02:52:39.134397 450 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:38361 (request call id 212) took 1758 ms (client timeout 1000 ms). Trace:
I20250112 02:52:39.134727 2047 consensus_queue.cc:1035] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
W20250112 02:52:39.134539 450 rpcz_store.cc:269] 0112 02:52:37.376140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.005323 (+629183us) service_pool.cc:225] Handling call
0112 02:52:38.310975 (+305652us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.310980 (+ 5us) write_op.cc:454] Released schema lock
0112 02:52:38.944261 (+633281us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x6130001737a0. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 000000000096ceda 00007f3e1053c4cd 00007f3e13205179 00007f3e1cc5e725 00007f3e1cbb5725 00007f3e1cbcb4fc 00007f3e1cd6f847 00007f3e13e72716 00007f3e13e72445 00007f3e105d130e 00007f3e105d01ee
0112 02:52:39.134358 (+190097us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:38.305114 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:38.305191 (+ 77us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:38.305196 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:38.305198 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:38.305321 (+ 123us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:38.305338 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:38.305342 (+ 4us) tablet.cc:682] Acquiring locks for 3 operations
0112 02:52:38.305427 (+ 85us) tablet.cc:698] Row locks acquired
0112 02:52:38.305429 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:38.305454 (+ 25us) write_op.cc:270] Start()
0112 02:52:38.305477 (+ 23us) write_op.cc:276] Timestamp: P: 1736650358305398 usec, L: 0
0112 02:52:38.305481 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:38.305636 (+ 155us) log.cc:844] Serialized 1275 byte log entry
0112 02:52:38.310003 (+ 4367us) op_driver.cc:464] REPLICATION: finished
0112 02:52:38.310131 (+ 128us) write_op.cc:301] APPLY: starting
0112 02:52:38.310141 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:38.310185 (+ 44us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:38.310212 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:38.310270 (+ 58us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:38.310294 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0112 02:52:38.310301 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:38.310353 (+ 52us) log.cc:844] Serialized 35 byte log entry
0112 02:52:38.310395 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.310409 (+ 14us) write_op.cc:454] Released schema lock
0112 02:52:38.310414 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":104041,"reported_tablets":216,"spinlock_wait_cycles":28326144,"child_traces":[["op",{"apply.queue_time_us":81,"apply.run_cpu_time_us":342,"apply.run_wall_time_us":342,"num_ops":3,"prepare.queue_time_us":36,"prepare.run_cpu_time_us":608,"prepare.run_wall_time_us":609,"raft.queue_time_us":53,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":192,"replication_time_us":4488}]]}
I20250112 02:52:39.119653 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 058a9ca089cd4967887b0a1127c7d03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:39.136929 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_5fb592e41cc4406aa27546e7b41fdb88" } new_table_name: "default.Table_41b7056b6e9240928d0af52aec2e9338" modify_external_catalogs: true
W20250112 02:52:39.138029 449 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:37987 (request call id 252) took 3374 ms (client timeout 1000 ms). Trace:
W20250112 02:52:39.139977 449 rpcz_store.cc:269] 0112 02:52:35.763589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:36.354736 (+591147us) service_pool.cc:225] Handling call
0112 02:52:36.401751 (+ 47015us) spinlock_profiling.cc:243] Waited 43.4 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd6d6b8 00007f3e13e72716 00007f3e13e72445 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:39.137979 (+2736228us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:38.248161 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:38.248243 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:38.248248 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:38.248250 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:38.248323 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:38.248335 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:38.248342 (+ 7us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:38.248385 (+ 43us) tablet.cc:698] Row locks acquired
0112 02:52:38.248387 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:38.248411 (+ 24us) write_op.cc:270] Start()
0112 02:52:38.248445 (+ 34us) write_op.cc:276] Timestamp: P: 1736650358248368 usec, L: 0
0112 02:52:38.248448 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:38.248563 (+ 115us) log.cc:844] Serialized 356 byte log entry
0112 02:52:38.250681 (+ 2118us) op_driver.cc:464] REPLICATION: finished
0112 02:52:38.250754 (+ 73us) write_op.cc:301] APPLY: starting
0112 02:52:38.250765 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:38.250790 (+ 25us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:38.250814 (+ 24us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:38.250848 (+ 34us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:38.250870 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:38.250876 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:38.250923 (+ 47us) log.cc:844] Serialized 23 byte log entry
0112 02:52:38.250963 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.250975 (+ 12us) write_op.cc:454] Released schema lock
0112 02:52:38.250979 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:38.251028 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.251030 (+ 2us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":162174,"reported_tablets":200,"spinlock_wait_cycles":99788032,"child_traces":[["op",{"apply.queue_time_us":42,"apply.run_cpu_time_us":266,"apply.run_wall_time_us":265,"num_ops":1,"prepare.queue_time_us":87,"prepare.run_cpu_time_us":483,"prepare.run_wall_time_us":484,"raft.queue_time_us":64,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":155,"replication_time_us":2204}]]}
I20250112 02:52:39.143183 1181 tablet_bootstrap.cc:492] T 709517ae520640208abe85fa14a5301e P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:39.143117 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_b46fcc60960d4de99ac8b7832afcb4b9" } modify_external_catalogs: true
I20250112 02:52:39.143299 1181 ts_tablet_manager.cc:1397] T 709517ae520640208abe85fa14a5301e P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.003s
I20250112 02:52:39.143427 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_b46fcc60960d4de99ac8b7832afcb4b9" } modify_external_catalogs: true
I20250112 02:52:39.143952 1181 raft_consensus.cc:357] T 709517ae520640208abe85fa14a5301e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.144121 1181 raft_consensus.cc:383] T 709517ae520640208abe85fa14a5301e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.144181 1181 raft_consensus.cc:738] T 709517ae520640208abe85fa14a5301e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.145793 1181 consensus_queue.cc:260] T 709517ae520640208abe85fa14a5301e P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:39.145639 447 rpcz_store.cc:269] 0112 02:52:36.392391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:36.463327 (+ 70936us) service_pool.cc:225] Handling call
0112 02:52:36.666857 (+203530us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd6d6b8 00007f3e13e72716 00007f3e13e72445 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:39.134068 (+2467211us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:37.594800 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:37.594927 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:37.594934 (+ 7us) write_op.cc:435] Acquired schema lock
0112 02:52:37.594937 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:37.595048 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:37.595061 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:37.595066 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:37.595170 (+ 104us) tablet.cc:698] Row locks acquired
0112 02:52:37.595173 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:37.595205 (+ 32us) write_op.cc:270] Start()
0112 02:52:37.595237 (+ 32us) write_op.cc:276] Timestamp: P: 1736650357595129 usec, L: 0
0112 02:52:37.595240 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:37.595422 (+ 182us) log.cc:844] Serialized 901 byte log entry
0112 02:52:37.599225 (+ 3803us) op_driver.cc:464] REPLICATION: finished
0112 02:52:37.599339 (+ 114us) write_op.cc:301] APPLY: starting
0112 02:52:37.599354 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:37.599397 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:37.599434 (+ 37us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:37.599542 (+ 108us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:37.599583 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:37.599590 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:37.599650 (+ 60us) log.cc:844] Serialized 29 byte log entry
0112 02:52:37.599701 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:37.599718 (+ 17us) write_op.cc:454] Released schema lock
0112 02:52:37.599724 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:37.599785 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:37.599789 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":34534,"reported_tablets":204,"spinlock_wait_cycles":456393984,"child_traces":[["op",{"apply.queue_time_us":67,"apply.run_cpu_time_us":443,"apply.run_wall_time_us":442,"num_ops":2,"prepare.queue_time_us":2752,"prepare.run_cpu_time_us":749,"prepare.run_wall_time_us":1100,"raft.queue_time_us":61,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":245,"replication_time_us":3957}]]}
I20250112 02:52:39.146749 1181 ts_tablet_manager.cc:1428] T 709517ae520640208abe85fa14a5301e P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:39.147013 1181 tablet_bootstrap.cc:492] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:39.148097 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_ab4d7c58924a4b8682b52637c23f125b" } new_table_name: "default.Table_8ab566cf9c8f45eba4dc9bb85264c823" modify_external_catalogs: true
I20250112 02:52:39.148820 1181 tablet_bootstrap.cc:654] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.149858 2523 tablet.cc:1721] T 5cc414d6420c41839621ab858067548b P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:52:39.150126 448 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:38361 (request call id 210) took 4871 ms (client timeout 1000 ms). Trace:
W20250112 02:52:39.150236 448 rpcz_store.cc:269] 0112 02:52:34.278260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.278324 (+ 64us) service_pool.cc:225] Handling call
0112 02:52:39.150084 (+4871760us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:35.033702 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:35.033799 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:35.033807 (+ 8us) write_op.cc:435] Acquired schema lock
0112 02:52:35.033811 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:35.033938 (+ 127us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:35.033950 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:35.033954 (+ 4us) tablet.cc:682] Acquiring locks for 3 operations
0112 02:52:35.034047 (+ 93us) tablet.cc:698] Row locks acquired
0112 02:52:35.034050 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:35.034079 (+ 29us) write_op.cc:270] Start()
0112 02:52:35.034107 (+ 28us) write_op.cc:276] Timestamp: P: 1736650355034063 usec, L: 0
0112 02:52:35.034110 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:35.034342 (+ 232us) log.cc:844] Serialized 1275 byte log entry
0112 02:52:35.038998 (+ 4656us) op_driver.cc:464] REPLICATION: finished
0112 02:52:35.039100 (+ 102us) write_op.cc:301] APPLY: starting
0112 02:52:35.039111 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:35.039191 (+ 80us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:35.039213 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:35.039306 (+ 93us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:35.039341 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0112 02:52:35.039348 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:35.039415 (+ 67us) log.cc:844] Serialized 35 byte log entry
0112 02:52:35.039934 (+ 519us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:35.039952 (+ 18us) write_op.cc:454] Released schema lock
0112 02:52:35.039958 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:35.040015 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:35.040019 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:36.327963 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:36.328063 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:36.328069 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:36.328072 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:36.328162 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:36.328171 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:36.328176 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:36.328239 (+ 63us) tablet.cc:698] Row locks acquired
0112 02:52:36.328242 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:36.328268 (+ 26us) write_op.cc:270] Start()
0112 02:52:36.328296 (+ 28us) write_op.cc:276] Timestamp: P: 1736650356328197 usec, L: 0
0112 02:52:36.328300 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:36.386649 (+ 58349us) log.cc:844] Serialized 356 byte log entry
0112 02:52:36.386846 (+ 197us) spinlock_profiling.cc:243] Waited 58.1 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 000000000096ceda 00007f3e150e3632 00007f3e1646beb5 00007f3e1646a9d9 00007f3e16472e00 00007f3e16472c0d 0000000000a71ecc 00007f3e098b0b64 00007f3e098b418b 00007f3e098b4008 0000000000a71ecc
0112 02:52:36.395033 (+ 8187us) spinlock_profiling.cc:243] Waited 6.87 ms on lock 0x616000042b48. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 000000000096ceda 00007f3e15090c23 00007f3e15091188 00007f3e15034d9c 00007f3e1500d8f7 00007f3e15015c4f 00007f3e1501593d 0000000000a71ecc 00007f3e098b0b64 00007f3e098b418b 00007f3e098b4008
0112 02:52:36.468806 (+ 73773us) op_driver.cc:464] REPLICATION: finished
0112 02:52:36.568142 (+ 99336us) write_op.cc:301] APPLY: starting
0112 02:52:36.568160 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:36.568209 (+ 49us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:36.568211 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:36.568263 (+ 52us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:36.568293 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:36.568300 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:36.568375 (+ 75us) log.cc:844] Serialized 23 byte log entry
0112 02:52:36.568444 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.568460 (+ 16us) write_op.cc:454] Released schema lock
0112 02:52:36.568466 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:36.568529 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:36.568535 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":19670,"reported_tablets":177,"spinlock_wait_cycles":669568,"child_traces":[["op",{"apply.queue_time_us":65,"apply.run_cpu_time_us":467,"apply.run_wall_time_us":907,"num_ops":3,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":735,"prepare.run_wall_time_us":736,"raft.queue_time_us":922,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":204,"replication_time_us":4866}],["op",{"apply.queue_time_us":99296,"apply.run_cpu_time_us":377,"apply.run_wall_time_us":378,"num_ops":1,"prepare.queue_time_us":575812,"prepare.run_cpu_time_us":1406,"prepare.run_wall_time_us":58963,"raft.queue_time_us":9366,"raft.run_cpu_time_us":597,"raft.run_wall_time_us":7369,"replication_time_us":140484,"spinlock_wait_cycles":149510272}]]}
I20250112 02:52:39.028916 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 49bb6d20cfa0453e90e048825114525e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:39.039716 989 consensus_peers.cc:487] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe614eaeb793473d9b2da206b7345562. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:52:39.152137 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 112447us
I20250112 02:52:39.152281 2024 log.cc:1198] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/1a8a178bafcb457f9b0e9587df560493
I20250112 02:52:39.152709 2024 ts_tablet_manager.cc:1939] T 1a8a178bafcb457f9b0e9587df560493 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:39.153093 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 73) took 4632 ms. Trace:
I20250112 02:52:39.042089 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003a3b4705e943c6a900762ec11ddd47" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:39.153307 1051 raft_consensus.cc:3054] T 003a3b4705e943c6a900762ec11ddd47 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.153362 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d5799b5b36487da42dc43b1c70ae26" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:39.153707 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 64707ee432fb4e96a3bbf7252dfa1813 with delete_type TABLET_DATA_DELETED (replaced by tablet b3c5a861a90840d0a33b780ec75bffd2) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.153913 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd498ccce484a89855c982762086890 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.154078 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet c04b64bc2f8348a086ed3c4e59f09b16 with delete_type TABLET_DATA_DELETED (replaced by tablet 61eebc5f22664712af6e74e82809ca87) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.154085 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f776e2a6ec704cf9b64483bc7d5b1dff" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:39.154668 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a3af0ea4e954053ae89814f3e74e6b7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:39.154891 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 97b31ebe035c4552baaffc3133ec73c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:24 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:39.084769 991 consensus_peers.cc:487] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4abf59d96414e5faa7cf75ba133832c. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:52:39.155555 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 70825.9us
I20250112 02:52:39.155447 2179 raft_consensus.cc:491] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.155750 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet fb0aba0139cf479b80b32f3dfc936535 with delete_type TABLET_DATA_DELETED (replaced by tablet e2f1f1a76e5f4308bf56dd647f02c6d8) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.155813 2179 raft_consensus.cc:513] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.155931 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d84027ad6d40a5be5afcdd854431a5 with delete_type TABLET_DATA_DELETED (replaced by tablet fbe9e9624ede40c29ea852b2b5494965) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.156189 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 04f135b226774285a077e5806131d91b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:39.156210 991 leader_election.cc:343] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.156332 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dda52f012794493a102ed1abc718f5b with delete_type TABLET_DATA_DELETED (replaced by tablet 17c54c5dc86446cf85974777c8145ef6) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.156417 2179 leader_election.cc:290] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.098439 2326 raft_consensus.cc:491] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.156715 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a18132549374218893f35a4e4226b72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.156847 2326 raft_consensus.cc:513] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.156927 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7320ac55dbfa47fe851fff94d32e1ffe" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:39.157073 910 raft_consensus.cc:2463] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:39.157119 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 058a9ca089cd4967887b0a1127c7d03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.157282 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d5b3df88a0c45dba879b3d506710614 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.157320 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 416fb6820cd54e118950c68d381f51b2) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.157430 2326 leader_election.cc:290] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.157400 713 leader_election.cc:304] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:39.157461 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet a85300a9aa1c4c50a3957a004bea7933 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.157614 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b184806b4884cd3b0a002905a1e57af with delete_type TABLET_DATA_DELETED (replaced by tablet 47a0d939fdb94590a86389f1f4693e9a) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.157658 2179 raft_consensus.cc:2798] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:39.157773 2179 raft_consensus.cc:491] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:39.157863 2179 raft_consensus.cc:3054] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.158021 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a7281c0c09442ebd0cc13b8496c86e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:39.158450 854 leader_election.cc:343] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.158558 854 leader_election.cc:304] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:39.158960 854 leader_election.cc:343] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.159030 854 leader_election.cc:304] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:39.159368 854 leader_election.cc:343] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.159457 854 leader_election.cc:304] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:39.159821 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet b8daf4ef3593410eae87672402292be6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.160269 2179 raft_consensus.cc:513] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.161091 2399 raft_consensus.cc:2743] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:39.161252 2399 raft_consensus.cc:2743] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:39.156719 991 leader_election.cc:343] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.161285 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7320ac55dbfa47fe851fff94d32e1ffe" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:39.161365 2399 raft_consensus.cc:2743] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:39.040659 1173 tablet_bootstrap.cc:492] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:39.161442 917 raft_consensus.cc:3054] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.161609 1173 ts_tablet_manager.cc:1397] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.185s user 0.006s sys 0.001s
W20250112 02:52:39.161862 991 consensus_peers.cc:487] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8daf4ef3593410eae87672402292be6. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:39.162334 2326 consensus_queue.cc:1035] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:39.162663 2326 consensus_queue.cc:1035] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:39.163877 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7320ac55dbfa47fe851fff94d32e1ffe" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:39.164024 1049 raft_consensus.cc:3054] T 7320ac55dbfa47fe851fff94d32e1ffe P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.164078 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 87759e8b99b04e628badc8f60cb38bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.164250 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c639a2bcd4740e4be7b2b162004669a with delete_type TABLET_DATA_DELETED (replaced by tablet 5d0c3191973a477e949fbf9dfcca963f) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.164021 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 61593969d2234ce3a6089027b067d922 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.164482 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.164723 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a3486ef011492c8a1df8d6278adedf" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:39.164920 771 raft_consensus.cc:2463] T c4a3486ef011492c8a1df8d6278adedf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:39.165391 1051 raft_consensus.cc:2463] T 003a3b4705e943c6a900762ec11ddd47 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:39.166154 1173 raft_consensus.cc:357] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.166360 1173 raft_consensus.cc:383] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.166416 1173 raft_consensus.cc:738] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.166502 1049 raft_consensus.cc:2463] T 7320ac55dbfa47fe851fff94d32e1ffe P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:39.166620 1173 consensus_queue.cc:260] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.167613 1173 ts_tablet_manager.cc:1428] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250112 02:52:39.168042 917 raft_consensus.cc:2463] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:39.168452 713 leader_election.cc:304] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:39.168725 2451 raft_consensus.cc:2798] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:39.168854 2451 raft_consensus.cc:695] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:39.168905 1173 tablet_bootstrap.cc:492] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:39.169163 2451 consensus_queue.cc:237] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.169461 991 leader_election.cc:304] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:39.169286 714 leader_election.cc:304] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:39.170531 2047 raft_consensus.cc:2798] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:39.170678 2047 raft_consensus.cc:695] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:39.170784 1173 tablet_bootstrap.cc:654] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.170955 2047 consensus_queue.cc:237] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.171576 2215 raft_consensus.cc:2798] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:39.171650 2215 raft_consensus.cc:491] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:39.171741 2215 raft_consensus.cc:3054] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.174104 2179 leader_election.cc:290] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.174480 2215 raft_consensus.cc:513] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.175127 2215 leader_election.cc:290] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.180285 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a3486ef011492c8a1df8d6278adedf" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:39.180497 779 raft_consensus.cc:3054] T c4a3486ef011492c8a1df8d6278adedf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.183676 779 raft_consensus.cc:2463] T c4a3486ef011492c8a1df8d6278adedf P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:39.184062 991 leader_election.cc:304] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:39.184345 2208 raft_consensus.cc:2798] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:39.184455 2208 raft_consensus.cc:695] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:39.184700 2208 consensus_queue.cc:237] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.189956 1177 tablet_bootstrap.cc:492] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:39.190125 1177 ts_tablet_manager.cc:1397] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.145s user 0.001s sys 0.019s
I20250112 02:52:39.191021 1177 raft_consensus.cc:357] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.191256 1177 raft_consensus.cc:383] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.191361 1177 raft_consensus.cc:738] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.184072 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7320ac55dbfa47fe851fff94d32e1ffe" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:39.199437 1050 raft_consensus.cc:2371] T 7320ac55dbfa47fe851fff94d32e1ffe P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1. Re-sending same reply.
I20250112 02:52:39.202584 1173 tablet_bootstrap.cc:492] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:39.202728 1173 ts_tablet_manager.cc:1397] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.000s
I20250112 02:52:39.203505 1173 raft_consensus.cc:357] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.203729 1173 raft_consensus.cc:383] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.203795 1173 raft_consensus.cc:738] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.203998 1173 consensus_queue.cc:260] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.191773 1177 consensus_queue.cc:260] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:39.210836 854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 51462.5us
W20250112 02:52:39.212002 989 leader_election.cc:343] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: c4a3486ef011492c8a1df8d6278adedf
W20250112 02:52:39.212440 989 leader_election.cc:343] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: c4a3486ef011492c8a1df8d6278adedf
I20250112 02:52:39.215117 2215 raft_consensus.cc:491] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.215308 2215 raft_consensus.cc:513] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.216135 2215 leader_election.cc:290] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.216682 2024 rpcz_store.cc:276] 0112 02:52:34.520202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.520420 (+ 218us) service_pool.cc:225] Handling call
0112 02:52:38.984937 (+4464517us) ts_tablet_manager.cc:1145] Deleting tablet 1a8a178bafcb457f9b0e9587df560493
0112 02:52:38.984948 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.012112 (+ 27164us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.014990 (+ 2878us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.153080 (+138090us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4442331}
I20250112 02:52:39.218364 2024 tablet_replica.cc:331] T 7a34d03a80ae402db6607261a5c40b60 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:39.218043 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 1a8a178bafcb457f9b0e9587df560493 (table default.Table_3c1f0242d17443b5a758967cf82ca0ca [id=02303baf5298474488dbf02c831f3d4d]) successfully deleted
I20250112 02:52:39.220216 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_5e7392cb5a68412f97bddb2677986918" } new_table_name: "default.Table_a9907ed8bfe14625977ec98b67e9d599" modify_external_catalogs: true
I20250112 02:52:39.215525 1177 ts_tablet_manager.cc:1428] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20250112 02:52:39.216434 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70715db0394d46d9b6b289160b00be7d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:39.222079 1177 tablet_bootstrap.cc:492] T 781fe3514f4c4f80b1ee49278502ccc1 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:39.222218 991 leader_election.cc:343] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.223270 2003 log.cc:1198] T 8ee98f055024429eafaa29bb961cab87 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/8ee98f055024429eafaa29bb961cab87
I20250112 02:52:39.225458 1173 ts_tablet_manager.cc:1428] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.023s user 0.000s sys 0.002s
I20250112 02:52:39.228263 1173 tablet_bootstrap.cc:492] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:39.229231 989 leader_election.cc:343] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 70715db0394d46d9b6b289160b00be7d
I20250112 02:52:39.233000 989 leader_election.cc:304] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:39.233731 2215 raft_consensus.cc:2743] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:39.228880 2024 raft_consensus.cc:2238] T 7a34d03a80ae402db6607261a5c40b60 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:39.234055 2024 raft_consensus.cc:2267] T 7a34d03a80ae402db6607261a5c40b60 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.234634 2024 ts_tablet_manager.cc:1905] T 7a34d03a80ae402db6607261a5c40b60 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:39.239488 989 consensus_peers.cc:487] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8daf4ef3593410eae87672402292be6. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:39.241914 2003 ts_tablet_manager.cc:1939] T 8ee98f055024429eafaa29bb961cab87 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:39.245982 2562 ts_tablet_manager.cc:1918] T 7e1954e7859f422f9690fe4a9384a8bb P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:39.246110 2562 log.cc:1198] T 7e1954e7859f422f9690fe4a9384a8bb P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/7e1954e7859f422f9690fe4a9384a8bb
I20250112 02:52:39.246454 2562 ts_tablet_manager.cc:1939] T 7e1954e7859f422f9690fe4a9384a8bb P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:39.255008 446 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 141) took 3324 ms. Trace:
I20250112 02:52:39.255319 446 rpcz_store.cc:276] 0112 02:52:35.930762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.105661 (+3174899us) service_pool.cc:225] Handling call
0112 02:52:39.122545 (+ 16884us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:39.122754 (+ 209us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:39.122919 (+ 165us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:39.122986 (+ 67us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:39.148668 (+ 25682us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:39.254965 (+106297us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:39.123369 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:39.123449 (+ 80us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:39.123456 (+ 7us) write_op.cc:435] Acquired schema lock
0112 02:52:39.123457 (+ 1us) tablet.cc:659] Decoding operations
0112 02:52:39.123562 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:39.123570 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:39.123574 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:39.123631 (+ 57us) tablet.cc:698] Row locks acquired
0112 02:52:39.123633 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:39.123662 (+ 29us) write_op.cc:270] Start()
0112 02:52:39.123690 (+ 28us) write_op.cc:276] Timestamp: P: 1736650359123543 usec, L: 0
0112 02:52:39.123695 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:39.123828 (+ 133us) log.cc:844] Serialized 405 byte log entry
0112 02:52:39.127724 (+ 3896us) op_driver.cc:464] REPLICATION: finished
0112 02:52:39.128453 (+ 729us) write_op.cc:301] APPLY: starting
0112 02:52:39.128465 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:39.128501 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:39.128582 (+ 81us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:39.128634 (+ 52us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:39.128668 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:39.128679 (+ 11us) write_op.cc:312] APPLY: finished
0112 02:52:39.128759 (+ 80us) log.cc:844] Serialized 23 byte log entry
0112 02:52:39.128801 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.128816 (+ 15us) write_op.cc:454] Released schema lock
0112 02:52:39.128824 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":692,"apply.run_cpu_time_us":428,"apply.run_wall_time_us":428,"num_ops":1,"prepare.queue_time_us":31,"prepare.run_cpu_time_us":527,"prepare.run_wall_time_us":526,"raft.queue_time_us":811,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":4012,"thread_start_us":5747,"threads_started":1}]]}
I20250112 02:52:39.257143 446 master_service.cc:946] ReplaceTablet: received request to replace tablet f31b777e7e0f43a28839d8c48c6a6590 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:39.260118 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 7e1954e7859f422f9690fe4a9384a8bb (table default.Table_829049fe22b74e62a87dd1bed65fb3fe [id=2e892eba68234ffca0be4ef3e49cf309]) successfully deleted
W20250112 02:52:39.260432 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet c29a9ca43e034b43a00207e5847bbee3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c29a9ca43e034b43a00207e5847bbee3 already in progress: creating tablet
I20250112 02:52:39.262318 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 106) took 4538 ms. Trace:
I20250112 02:52:39.263495 2003 rpcz_store.cc:276] 0112 02:52:34.704287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.704332 (+ 45us) service_pool.cc:225] Handling call
0112 02:52:38.957128 (+4252796us) ts_tablet_manager.cc:1145] Deleting tablet 8ee98f055024429eafaa29bb961cab87
0112 02:52:38.957137 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.096590 (+139453us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.113654 (+ 17064us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.242301 (+128647us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34267,"tablet-delete.queue_time_us":4251512}
I20250112 02:52:39.263953 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 8ee98f055024429eafaa29bb961cab87 (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]) successfully deleted
W20250112 02:52:39.264526 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet a8e7882e07784bdda18dc1ffd20963c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8e7882e07784bdda18dc1ffd20963c8 already in progress: creating tablet
I20250112 02:52:39.264756 1173 tablet_bootstrap.cc:654] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.278357 1177 tablet_bootstrap.cc:654] T 781fe3514f4c4f80b1ee49278502ccc1 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.279546 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 107) took 4558 ms. Trace:
I20250112 02:52:39.284493 2003 rpcz_store.cc:276] 0112 02:52:34.705858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.705894 (+ 36us) service_pool.cc:225] Handling call
0112 02:52:39.263924 (+4558030us) ts_tablet_manager.cc:1145] Deleting tablet c29a9ca43e034b43a00207e5847bbee3
0112 02:52:39.263934 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.264019 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4553819}
W20250112 02:52:39.284894 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet c29a9ca43e034b43a00207e5847bbee3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c29a9ca43e034b43a00207e5847bbee3 already in progress: creating tablet
I20250112 02:52:39.284008 2179 consensus_queue.cc:1035] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20250112 02:52:39.285806 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 108) took 4417 ms. Trace:
I20250112 02:52:39.285902 2003 rpcz_store.cc:276] 0112 02:52:34.879247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.883678 (+ 4431us) service_pool.cc:225] Handling call
0112 02:52:39.284895 (+4401217us) ts_tablet_manager.cc:1145] Deleting tablet 83c04f1a2cb04950a9f6170e252c19b4
0112 02:52:39.284903 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.284974 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4400889}
W20250112 02:52:39.286397 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 83c04f1a2cb04950a9f6170e252c19b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83c04f1a2cb04950a9f6170e252c19b4 already in progress: creating tablet
I20250112 02:52:39.287246 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 146) took 3555 ms. Trace:
I20250112 02:52:39.287339 2003 rpcz_store.cc:276] 0112 02:52:35.731390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:35.731437 (+ 47us) service_pool.cc:225] Handling call
0112 02:52:39.286384 (+3554947us) ts_tablet_manager.cc:1145] Deleting tablet 82b351c563d749df806ab5d0331a4ef2
0112 02:52:39.286392 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.286459 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3554733}
W20250112 02:52:39.287890 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 82b351c563d749df806ab5d0331a4ef2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82b351c563d749df806ab5d0331a4ef2 already in progress: creating tablet
I20250112 02:52:39.289897 2179 consensus_queue.cc:1035] T f0dc78d722ac4687bfb3dce94d6db7f0 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:39.292335 2525 tablet.cc:1721] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:39.293535 2024 ts_tablet_manager.cc:1918] T 7a34d03a80ae402db6607261a5c40b60 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:39.295251 1181 tablet_bootstrap.cc:492] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:39.295416 1181 ts_tablet_manager.cc:1397] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.148s user 0.010s sys 0.003s
I20250112 02:52:39.296134 1181 raft_consensus.cc:357] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.296345 1181 raft_consensus.cc:383] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.296418 1181 raft_consensus.cc:738] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.296648 1181 consensus_queue.cc:260] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.297482 1181 ts_tablet_manager.cc:1428] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:39.297847 2024 log.cc:1198] T 7a34d03a80ae402db6607261a5c40b60 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/7a34d03a80ae402db6607261a5c40b60
I20250112 02:52:39.297922 2003 tablet_replica.cc:331] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:39.297951 2523 tablet.cc:1721] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:39.298122 2003 raft_consensus.cc:2238] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:39.298199 2024 ts_tablet_manager.cc:1939] T 7a34d03a80ae402db6607261a5c40b60 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:39.298703 2522 tablet.cc:1721] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:39.300330 1102 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:39.303546 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 75) took 4783 ms. Trace:
I20250112 02:52:39.305727 2179 raft_consensus.cc:491] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.305852 2179 raft_consensus.cc:513] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.305881 2024 rpcz_store.cc:276] 0112 02:52:34.520380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.520502 (+ 122us) service_pool.cc:225] Handling call
0112 02:52:39.218304 (+4697802us) ts_tablet_manager.cc:1145] Deleting tablet 7a34d03a80ae402db6607261a5c40b60
0112 02:52:39.218319 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.254805 (+ 36486us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.293433 (+ 38628us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.303529 (+ 10096us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10184,"tablet-delete.queue_time_us":4675642}
I20250112 02:52:39.306354 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 7a34d03a80ae402db6607261a5c40b60 (table default.Table_af61980bca5e4995a77da652dcfb3f0d [id=9d7284a74b4148c0a8d20eb220ef9e17]) successfully deleted
I20250112 02:52:39.306704 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a34d03a80ae402db6607261a5c40b60" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:39.306962 918 raft_consensus.cc:2463] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:39.307267 2562 tablet_replica.cc:331] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:39.307705 2598 raft_consensus.cc:491] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.307824 2598 raft_consensus.cc:513] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.308393 2598 leader_election.cc:290] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:39.309247 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54215c55966a46e1ab96e352c4e3764e" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:39.309844 854 leader_election.cc:343] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.310132 2179 leader_election.cc:290] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.310104 713 leader_election.cc:304] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:39.310065 2562 raft_consensus.cc:2238] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:39.310352 2179 raft_consensus.cc:2798] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:39.310359 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54215c55966a46e1ab96e352c4e3764e" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:39.310602 2179 raft_consensus.cc:491] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:39.310729 2179 raft_consensus.cc:3054] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:39.310806 714 leader_election.cc:343] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 7a34d03a80ae402db6607261a5c40b60
W20250112 02:52:39.310904 855 leader_election.cc:343] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.310988 855 leader_election.cc:304] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:39.311281 2598 raft_consensus.cc:2743] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:39.311725 2562 raft_consensus.cc:2267] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.311686 2208 consensus_queue.cc:1035] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:39.313040 2562 ts_tablet_manager.cc:1905] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.313946 1173 tablet_bootstrap.cc:492] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:39.314160 1173 ts_tablet_manager.cc:1397] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.088s user 0.002s sys 0.004s
I20250112 02:52:39.314488 1181 tablet_bootstrap.cc:492] T a3ffdc237f8640f0b378809f93fc6292 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:39.315022 1173 raft_consensus.cc:357] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.315248 1173 raft_consensus.cc:383] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.315321 1173 raft_consensus.cc:738] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.315615 1173 consensus_queue.cc:260] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.316473 1173 ts_tablet_manager.cc:1428] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:39.317718 1181 tablet_bootstrap.cc:654] T a3ffdc237f8640f0b378809f93fc6292 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.317436 1173 tablet_bootstrap.cc:492] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:39.317940 965 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:39.319639 2179 raft_consensus.cc:513] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.320793 2179 leader_election.cc:290] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:39.323930 714 leader_election.cc:343] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 7a34d03a80ae402db6607261a5c40b60
I20250112 02:52:39.327554 2003 pending_rounds.cc:62] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:39.327682 2003 pending_rounds.cc:69] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319871166554112 op_type: NO_OP noop_request { }
I20250112 02:52:39.327807 2003 raft_consensus.cc:2883] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:39.327850 2024 tablet_replica.cc:331] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:39.327906 2003 raft_consensus.cc:2267] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.328132 2024 raft_consensus.cc:2238] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:39.328439 2024 raft_consensus.cc:2267] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.334245 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet f31b777e7e0f43a28839d8c48c6a6590
I20250112 02:52:39.336148 2003 ts_tablet_manager.cc:1905] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.339586 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a34d03a80ae402db6607261a5c40b60" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:39.339823 917 raft_consensus.cc:3054] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.335599 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet f31b777e7e0f43a28839d8c48c6a6590 with delete_type TABLET_DATA_DELETED (replaced by tablet ca2165362ee541adac724b63f39b1489) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:39.340490 2024 ts_tablet_manager.cc:1905] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.342097 917 raft_consensus.cc:2463] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:39.342505 713 leader_election.cc:304] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:39.342825 2047 raft_consensus.cc:2798] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:39.343008 2047 raft_consensus.cc:695] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:39.343593 747 tablet_service.cc:1514] Processing DeleteTablet for tablet f31b777e7e0f43a28839d8c48c6a6590 with delete_type TABLET_DATA_DELETED (replaced by tablet ca2165362ee541adac724b63f39b1489) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:39.343835 895 tablet_service.cc:1514] Processing DeleteTablet for tablet f31b777e7e0f43a28839d8c48c6a6590 with delete_type TABLET_DATA_DELETED (replaced by tablet ca2165362ee541adac724b63f39b1489) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:39.344238 446 catalog_manager.cc:6596] ReplaceTablet: tablet f31b777e7e0f43a28839d8c48c6a6590 deleted and replaced by tablet ca2165362ee541adac724b63f39b1489
I20250112 02:52:39.344302 2047 consensus_queue.cc:237] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.344421 446 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:40610 (request call id 147) took 3179 ms. Trace:
I20250112 02:52:39.344530 446 rpcz_store.cc:276] 0112 02:52:36.165402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.257069 (+3091667us) service_pool.cc:225] Handling call
0112 02:52:39.344374 (+ 87305us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:39.258347 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:39.258431 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:39.258437 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:39.258439 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:39.258530 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:39.258541 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:39.258546 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:39.258613 (+ 67us) tablet.cc:698] Row locks acquired
0112 02:52:39.258616 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:39.258655 (+ 39us) write_op.cc:270] Start()
0112 02:52:39.258684 (+ 29us) write_op.cc:276] Timestamp: P: 1736650359258613 usec, L: 0
0112 02:52:39.258687 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:39.258850 (+ 163us) log.cc:844] Serialized 656 byte log entry
0112 02:52:39.326539 (+ 67689us) op_driver.cc:464] REPLICATION: finished
0112 02:52:39.327064 (+ 525us) write_op.cc:301] APPLY: starting
0112 02:52:39.327072 (+ 8us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:39.327099 (+ 27us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:39.327116 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:39.327183 (+ 67us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:39.327223 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:39.327229 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:39.327268 (+ 39us) log.cc:844] Serialized 29 byte log entry
0112 02:52:39.327299 (+ 31us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.327311 (+ 12us) write_op.cc:454] Released schema lock
0112 02:52:39.327315 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:39.327383 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.327386 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":486,"apply.run_cpu_time_us":309,"apply.run_wall_time_us":309,"num_ops":2,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":562,"prepare.run_wall_time_us":562,"replication_time_us":67840}]]}
I20250112 02:52:39.336189 1173 tablet_bootstrap.cc:654] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.350910 2003 ts_tablet_manager.cc:1918] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:39.351027 2003 log.cc:1198] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/003a3b4705e943c6a900762ec11ddd47
I20250112 02:52:39.351364 2003 ts_tablet_manager.cc:1939] T 003a3b4705e943c6a900762ec11ddd47 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:39.352541 2562 ts_tablet_manager.cc:1918] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:39.352679 2562 log.cc:1198] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/bab4978edea0445fb0456e88e7fb4c44
I20250112 02:52:39.353022 2562 ts_tablet_manager.cc:1939] T bab4978edea0445fb0456e88e7fb4c44 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:39.356859 2598 raft_consensus.cc:491] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.357021 2598 raft_consensus.cc:513] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.357328 1177 tablet_bootstrap.cc:492] T 781fe3514f4c4f80b1ee49278502ccc1 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:39.357460 1177 ts_tablet_manager.cc:1397] T 781fe3514f4c4f80b1ee49278502ccc1 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.135s user 0.003s sys 0.003s
I20250112 02:52:39.357599 2598 leader_election.cc:290] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:39.357698 1181 tablet_bootstrap.cc:492] T a3ffdc237f8640f0b378809f93fc6292 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:39.357863 1181 ts_tablet_manager.cc:1397] T a3ffdc237f8640f0b378809f93fc6292 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.001s
W20250112 02:52:39.358090 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet f3771166b77e4f90aaa116ab1a9dc299 (table default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f3771166b77e4f90aaa116ab1a9dc299 consensus state has no leader
I20250112 02:52:39.358249 1177 raft_consensus.cc:357] T 781fe3514f4c4f80b1ee49278502ccc1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.358362 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26577b614e32458e8bc5d6830c8c2e40" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:39.358464 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26577b614e32458e8bc5d6830c8c2e40" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:39.358493 2326 consensus_queue.cc:1035] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:52:39.358899 855 leader_election.cc:343] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.359123 1177 raft_consensus.cc:383] T 781fe3514f4c4f80b1ee49278502ccc1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.359215 1177 raft_consensus.cc:738] T 781fe3514f4c4f80b1ee49278502ccc1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
W20250112 02:52:39.359237 854 leader_election.cc:343] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.359334 854 leader_election.cc:304] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:39.359453 1177 consensus_queue.cc:260] T 781fe3514f4c4f80b1ee49278502ccc1 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.359596 2598 raft_consensus.cc:2743] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:39.360255 1177 ts_tablet_manager.cc:1428] T 781fe3514f4c4f80b1ee49278502ccc1 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 02:52:39.360600 1177 tablet_bootstrap.cc:492] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:39.362557 1177 tablet_bootstrap.cc:654] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.363914 2451 raft_consensus.cc:491] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.364060 2451 raft_consensus.cc:513] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.364737 2451 leader_election.cc:290] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.365658 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d84027ad6d40a5be5afcdd854431a5" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:39.365903 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d84027ad6d40a5be5afcdd854431a5" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:39.366099 910 raft_consensus.cc:2463] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
W20250112 02:52:39.366113 714 leader_election.cc:343] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.366432 713 leader_election.cc:304] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:39.366834 2451 raft_consensus.cc:2798] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:39.366945 2451 raft_consensus.cc:491] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:39.366946 2179 consensus_queue.cc:1035] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:39.367062 2451 raft_consensus.cc:3054] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.373059 1181 raft_consensus.cc:357] T a3ffdc237f8640f0b378809f93fc6292 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.377166 1181 raft_consensus.cc:383] T a3ffdc237f8640f0b378809f93fc6292 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.377279 1181 raft_consensus.cc:738] T a3ffdc237f8640f0b378809f93fc6292 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.376670 2562 tablet_replica.cc:331] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:39.377789 2562 raft_consensus.cc:2238] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:39.377976 2562 raft_consensus.cc:2267] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.378449 2562 ts_tablet_manager.cc:1905] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.379864 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 149) took 3631 ms. Trace:
I20250112 02:52:39.380028 2003 rpcz_store.cc:276] 0112 02:52:35.748206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:35.748568 (+ 362us) service_pool.cc:225] Handling call
0112 02:52:39.287795 (+3539227us) ts_tablet_manager.cc:1145] Deleting tablet 003a3b4705e943c6a900762ec11ddd47
0112 02:52:39.288310 (+ 515us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.338679 (+ 50369us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.350818 (+ 12139us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.379809 (+ 28991us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3523810}
I20250112 02:52:39.380681 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 165) took 1600 ms. Trace:
I20250112 02:52:39.380822 2003 rpcz_store.cc:276] 0112 02:52:37.779892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:37.780034 (+ 142us) service_pool.cc:225] Handling call
0112 02:52:39.380520 (+1600486us) ts_tablet_manager.cc:1145] Deleting tablet 13578542e9e741e8817e0ee63b10f1a1
0112 02:52:39.380532 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.380619 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1600298}
I20250112 02:52:39.381392 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 166) took 1366 ms. Trace:
I20250112 02:52:39.381421 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet bab4978edea0445fb0456e88e7fb4c44 (table default.Table_9dd5993a01b44a0abe33a5738c8d5c2b [id=6a7dcf7429b24699b036c53b3005051f]) successfully deleted
I20250112 02:52:39.381536 2003 rpcz_store.cc:276] 0112 02:52:38.014568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.014623 (+ 55us) service_pool.cc:225] Handling call
0112 02:52:39.381240 (+1366617us) ts_tablet_manager.cc:1145] Deleting tablet a3ffdc237f8640f0b378809f93fc6292
0112 02:52:39.381250 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.381337 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1366418}
W20250112 02:52:39.381711 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 8c821ea3ca4946c290c17135366e0cac because the tablet was not found. No further retry: Not found: Tablet not found: 8c821ea3ca4946c290c17135366e0cac
I20250112 02:52:39.381989 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 003a3b4705e943c6a900762ec11ddd47 (table default.Table_743b3f7029534564b35c0c137b35fd75 [id=b2fa3922f6d447769a8179052f9a1861]) successfully deleted
I20250112 02:52:39.382046 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 167) took 1115 ms. Trace:
I20250112 02:52:39.382136 2003 rpcz_store.cc:276] 0112 02:52:38.267000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.267237 (+ 237us) service_pool.cc:225] Handling call
0112 02:52:39.381890 (+1114653us) ts_tablet_manager.cc:1145] Deleting tablet cbe30f9860e2492b90e8cedf8ff9daba
0112 02:52:39.381911 (+ 21us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.381988 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1114441}
W20250112 02:52:39.382347 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 13578542e9e741e8817e0ee63b10f1a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13578542e9e741e8817e0ee63b10f1a1 already in progress: creating tablet
W20250112 02:52:39.382535 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet a3ffdc237f8640f0b378809f93fc6292 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3ffdc237f8640f0b378809f93fc6292 already in progress: creating tablet
W20250112 02:52:39.382737 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet cbe30f9860e2492b90e8cedf8ff9daba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbe30f9860e2492b90e8cedf8ff9daba already in progress: creating tablet
W20250112 02:52:39.382997 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d63d411f906b4d4eb68343bdcc493e05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d63d411f906b4d4eb68343bdcc493e05 already in progress: creating tablet
W20250112 02:52:39.383184 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 7e1954e7859f422f9690fe4a9384a8bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e1954e7859f422f9690fe4a9384a8bb already in progress: creating tablet
I20250112 02:52:39.383220 1181 consensus_queue.cc:260] T a3ffdc237f8640f0b378809f93fc6292 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:39.383390 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet bd5b3c712c0f448ea9b14ab93dd731ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd5b3c712c0f448ea9b14ab93dd731ab already in progress: creating tablet
W20250112 02:52:39.383901 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet bb217afc08304c0ea35dee3120347ba1 because the tablet was not found. No further retry: Not found: Tablet not found: bb217afc08304c0ea35dee3120347ba1
W20250112 02:52:39.384080 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 4c0c2fad4f3243238641aae56b305bb5 because the tablet was not found. No further retry: Not found: Tablet not found: 4c0c2fad4f3243238641aae56b305bb5
W20250112 02:52:39.384236 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 02d0f67141dc46b6a87ca0ad080d5bfe because the tablet was not found. No further retry: Not found: Tablet not found: 02d0f67141dc46b6a87ca0ad080d5bfe
W20250112 02:52:39.384442 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d659c5a4f7a446c6aaf2283f65b45106 because the tablet was not found. No further retry: Not found: Tablet not found: d659c5a4f7a446c6aaf2283f65b45106
I20250112 02:52:39.384811 2451 raft_consensus.cc:513] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.385913 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d84027ad6d40a5be5afcdd854431a5" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:39.386445 714 leader_election.cc:343] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.387199 2451 leader_election.cc:290] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.387493 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d84027ad6d40a5be5afcdd854431a5" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:39.387739 910 raft_consensus.cc:3054] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.390177 910 raft_consensus.cc:2463] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:39.390774 966 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:39.390743 713 leader_election.cc:304] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:39.391115 2451 raft_consensus.cc:2798] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:39.391309 2451 raft_consensus.cc:695] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:39.391645 2451 consensus_queue.cc:237] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.393906 1181 ts_tablet_manager.cc:1428] T a3ffdc237f8640f0b378809f93fc6292 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.036s user 0.001s sys 0.001s
I20250112 02:52:39.394191 1181 tablet_bootstrap.cc:492] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:39.395788 2003 tablet_replica.cc:331] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:39.396049 2003 raft_consensus.cc:2238] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:39.398068 1173 tablet_bootstrap.cc:492] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:39.398195 1173 ts_tablet_manager.cc:1397] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.081s user 0.005s sys 0.002s
I20250112 02:52:39.398339 2003 raft_consensus.cc:2267] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.398792 1173 raft_consensus.cc:357] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.398976 1173 raft_consensus.cc:383] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.399032 1173 raft_consensus.cc:738] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.399236 1173 consensus_queue.cc:260] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.399799 1173 ts_tablet_manager.cc:1428] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:39.399832 2003 ts_tablet_manager.cc:1905] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:39.406162 991 consensus_peers.cc:487] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ee98f055024429eafaa29bb961cab87. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:39.406435 2326 consensus_queue.cc:1035] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:39.406539 2024 ts_tablet_manager.cc:1918] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:39.406611 2024 log.cc:1198] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/ea33e5c1a1fb46599c5836a36b130051
I20250112 02:52:39.406951 2024 ts_tablet_manager.cc:1939] T ea33e5c1a1fb46599c5836a36b130051 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:39.410398 1177 tablet_bootstrap.cc:492] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:39.410758 1177 ts_tablet_manager.cc:1397] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.003s
I20250112 02:52:39.411672 1181 tablet_bootstrap.cc:654] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.411666 1177 raft_consensus.cc:357] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.412799 1177 raft_consensus.cc:383] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.413057 1177 raft_consensus.cc:738] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.413381 1177 consensus_queue.cc:260] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.414225 1177 ts_tablet_manager.cc:1428] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 02:52:39.414352 1173 tablet_bootstrap.cc:492] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:39.414964 1177 tablet_bootstrap.cc:492] T f924459d1150472aafb325a11825db23 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:39.415602 1173 tablet_bootstrap.cc:654] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.417076 1177 tablet_bootstrap.cc:654] T f924459d1150472aafb325a11825db23 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.418464 2562 ts_tablet_manager.cc:1918] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:39.427160 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 76) took 4906 ms. Trace:
I20250112 02:52:39.427691 2024 rpcz_store.cc:276] 0112 02:52:34.520464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.521848 (+ 1384us) service_pool.cc:225] Handling call
0112 02:52:39.327744 (+4805896us) ts_tablet_manager.cc:1145] Deleting tablet ea33e5c1a1fb46599c5836a36b130051
0112 02:52:39.327752 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.347339 (+ 19587us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.406460 (+ 59121us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.426993 (+ 20533us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4785063}
I20250112 02:52:39.429096 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet ea33e5c1a1fb46599c5836a36b130051 (table default.Table_7abcf14838c744109a08985ba6ce3e0d [id=5f25c42be8924b41be7f2f268d068e04]) successfully deleted
I20250112 02:52:39.432284 2024 tablet_replica.cc:331] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:39.432624 2024 raft_consensus.cc:2238] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:39.433173 2024 pending_rounds.cc:62] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6: Trying to abort 1 pending ops.
I20250112 02:52:39.433277 2024 pending_rounds.cc:69] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319849634189312 op_type: NO_OP noop_request { }
I20250112 02:52:39.433508 2024 raft_consensus.cc:2883] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:39.433871 2024 raft_consensus.cc:2267] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.438022 2024 ts_tablet_manager.cc:1905] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.446625 1173 tablet_bootstrap.cc:492] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:39.446741 1173 ts_tablet_manager.cc:1397] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.005s
I20250112 02:52:39.446977 1181 tablet_bootstrap.cc:492] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:39.447137 1181 ts_tablet_manager.cc:1397] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.001s
I20250112 02:52:39.447885 1181 raft_consensus.cc:357] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.448119 1181 raft_consensus.cc:383] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.448207 1181 raft_consensus.cc:738] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.449214 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:39.450709 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:39.451370 2451 raft_consensus.cc:491] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.451539 2451 raft_consensus.cc:513] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.452450 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e378a6a9b4a455c9f5e1d1ec0fd6ca0" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:39.452627 1050 raft_consensus.cc:2463] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:39.453090 714 leader_election.cc:304] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:39.453337 2047 raft_consensus.cc:2798] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:39.453424 2047 raft_consensus.cc:491] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:39.453492 2047 raft_consensus.cc:3054] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.455762 2047 raft_consensus.cc:513] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.455981 2024 ts_tablet_manager.cc:1918] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:39.456130 2024 log.cc:1198] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/bb217afc08304c0ea35dee3120347ba1
I20250112 02:52:39.456765 2024 ts_tablet_manager.cc:1939] T bb217afc08304c0ea35dee3120347ba1 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:39.457114 1177 tablet_bootstrap.cc:492] T f924459d1150472aafb325a11825db23 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:39.457247 1177 ts_tablet_manager.cc:1397] T f924459d1150472aafb325a11825db23 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
I20250112 02:52:39.457378 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 77) took 4935 ms. Trace:
I20250112 02:52:39.457484 2024 rpcz_store.cc:276] 0112 02:52:34.522172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.532382 (+ 10210us) service_pool.cc:225] Handling call
0112 02:52:39.429919 (+4897537us) ts_tablet_manager.cc:1145] Deleting tablet bb217afc08304c0ea35dee3120347ba1
0112 02:52:39.432228 (+ 2309us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.451266 (+ 19038us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.455811 (+ 4545us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.457361 (+ 1550us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4887034}
I20250112 02:52:39.457892 1177 raft_consensus.cc:357] T f924459d1150472aafb325a11825db23 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:39.458024 2024 tablet_replica.cc:331] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:39.458077 1177 raft_consensus.cc:383] T f924459d1150472aafb325a11825db23 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.458139 1177 raft_consensus.cc:738] T f924459d1150472aafb325a11825db23 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.458134 2215 raft_consensus.cc:491] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.458226 2215 raft_consensus.cc:513] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.458287 2024 raft_consensus.cc:2238] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:39.458326 1177 consensus_queue.cc:260] T f924459d1150472aafb325a11825db23 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:39.458525 1101 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:39.458485 2024 raft_consensus.cc:2267] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.458702 2215 leader_election.cc:290] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.459142 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af55dab202444358ea1a0674d37946d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:39.459517 2024 ts_tablet_manager.cc:1905] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:39.459565 991 leader_election.cc:343] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.457973 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet bb217afc08304c0ea35dee3120347ba1 (table default.Table_94d3a2be9a344af082b69ecfa503d4c0 [id=da2e1bf15b21475085a3da6b16aecc4c]) successfully deleted
I20250112 02:52:39.459837 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af55dab202444358ea1a0674d37946d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:39.459990 917 raft_consensus.cc:2463] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:39.460294 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e378a6a9b4a455c9f5e1d1ec0fd6ca0" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:39.460428 1050 raft_consensus.cc:3054] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.460569 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:39.460784 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:39.462733 1050 raft_consensus.cc:2463] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:39.463234 714 leader_election.cc:304] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:39.463598 2598 raft_consensus.cc:491] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.463574 989 leader_election.cc:304] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:39.463703 2598 raft_consensus.cc:513] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.463835 2215 raft_consensus.cc:2798] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:39.463891 2215 raft_consensus.cc:491] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:39.463939 2215 raft_consensus.cc:3054] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.464113 2179 raft_consensus.cc:2798] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:39.464236 2179 raft_consensus.cc:695] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:39.464243 2598 leader_election.cc:290] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:39.464428 2179 consensus_queue.cc:237] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.465821 2215 raft_consensus.cc:513] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.466289 2215 leader_election.cc:290] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.466986 2047 leader_election.cc:290] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.470237 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af55dab202444358ea1a0674d37946d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:39.470423 917 raft_consensus.cc:3054] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.470480 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7717cdcb8a4f47b11ba7af54b57e3c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:39.470507 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7717cdcb8a4f47b11ba7af54b57e3c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:39.471607 855 leader_election.cc:343] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.471940 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af55dab202444358ea1a0674d37946d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
W20250112 02:52:39.472374 991 leader_election.cc:343] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.473368 917 raft_consensus.cc:2463] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:39.473605 2451 leader_election.cc:290] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.473753 1181 consensus_queue.cc:260] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:39.473897 854 leader_election.cc:343] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.474001 854 leader_election.cc:304] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:39.474467 2598 raft_consensus.cc:2743] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:39.474771 989 leader_election.cc:304] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:39.475284 2326 raft_consensus.cc:2798] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:39.475392 2326 raft_consensus.cc:695] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:39.475637 2326 consensus_queue.cc:237] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.476071 1177 ts_tablet_manager.cc:1428] T f924459d1150472aafb325a11825db23 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250112 02:52:39.476325 1177 tablet_bootstrap.cc:492] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:39.476954 1181 ts_tablet_manager.cc:1428] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.030s user 0.001s sys 0.000s
I20250112 02:52:39.477188 1181 tablet_bootstrap.cc:492] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:39.477739 1177 tablet_bootstrap.cc:654] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.477779 2598 raft_consensus.cc:491] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.477890 2598 raft_consensus.cc:513] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.478379 2598 leader_election.cc:290] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:39.479007 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8429a3d8636d4e2ca4c1191f85b9266b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:39.479401 855 leader_election.cc:343] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.479763 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8429a3d8636d4e2ca4c1191f85b9266b" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:39.479835 2228 raft_consensus.cc:491] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.479954 2228 raft_consensus.cc:513] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.480069 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e378a6a9b4a455c9f5e1d1ec0fd6ca0" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:39.480212 910 raft_consensus.cc:3054] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:39.480269 854 leader_election.cc:343] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.480338 854 leader_election.cc:304] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:39.480458 2228 leader_election.cc:290] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.481135 2598 raft_consensus.cc:2743] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:39.481361 989 leader_election.cc:343] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: b9c006281dc949119951c74e23167d7f
I20250112 02:52:39.481523 778 raft_consensus.cc:3054] T e4d42f1611de49d6b6e5aaa9be907054 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.481719 1173 raft_consensus.cc:357] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.481874 1173 raft_consensus.cc:383] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.481928 1173 raft_consensus.cc:738] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.482092 1173 consensus_queue.cc:260] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.482517 1173 ts_tablet_manager.cc:1428] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.036s user 0.000s sys 0.001s
I20250112 02:52:39.482723 1173 tablet_bootstrap.cc:492] T afd6d31155034908ac44a90c3a2ea7d6 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:39.482766 910 raft_consensus.cc:2463] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:39.483392 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e378a6a9b4a455c9f5e1d1ec0fd6ca0" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:39.483553 917 raft_consensus.cc:2371] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1. Re-sending same reply.
W20250112 02:52:39.485108 991 leader_election.cc:343] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: b9c006281dc949119951c74e23167d7f
I20250112 02:52:39.485230 991 leader_election.cc:304] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:39.485720 2228 raft_consensus.cc:2743] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:39.485875 2228 consensus_queue.cc:1035] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:39.487711 2228 consensus_queue.cc:1035] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:39.493201 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_5d473cabff2e46e0b531fbce28d9d558 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 1 got 0
I20250112 02:52:39.493516 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:39.494735 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:39.495019 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_5e7392cb5a68412f97bddb2677986918 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 1 got 0
I20250112 02:52:39.495416 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 09d06e2e8c13464db8cf801a292bdc99 (table default.Table_2c52fead633649dd869763e5aa532be7 [id=e3bb489c53454222a79702b798c0c2f1]). Expected version 3 got 2
I20250112 02:52:39.495671 2562 log.cc:1198] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/b1d9fae290ec4281839991f9d40b105c
I20250112 02:52:39.498489 2003 ts_tablet_manager.cc:1918] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:39.498623 2003 log.cc:1198] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/416fb6820cd54e118950c68d381f51b2
I20250112 02:52:39.498936 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 1 got 0
I20250112 02:52:39.499066 2003 ts_tablet_manager.cc:1939] T 416fb6820cd54e118950c68d381f51b2 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:39.499478 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:39.500773 2003 tablet_replica.cc:331] T 87759e8b99b04e628badc8f60cb38bb3 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:39.501075 2003 raft_consensus.cc:2238] T 87759e8b99b04e628badc8f60cb38bb3 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:39.505092 1173 tablet_bootstrap.cc:654] T afd6d31155034908ac44a90c3a2ea7d6 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.500047 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 416fb6820cd54e118950c68d381f51b2 (table default.Table_f4d612cd6a524bc493147493c5a4a284 [id=4ac16bc62bbe4ed7a977d22c92ad580d]) successfully deleted
W20250112 02:52:39.505903 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 161fe26b9c0e4543b7aeb0021506d72a because the tablet was not found. No further retry: Not found: Tablet not found: 161fe26b9c0e4543b7aeb0021506d72a
W20250112 02:52:39.506137 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f7cd0927c6a4401cbe69f55d4db9963a because the tablet was not found. No further retry: Not found: Tablet not found: f7cd0927c6a4401cbe69f55d4db9963a
W20250112 02:52:39.506306 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet aadb2ab1d2024f889841ac91e065e064 because the tablet was not found. No further retry: Not found: Tablet not found: aadb2ab1d2024f889841ac91e065e064
W20250112 02:52:39.506489 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 1c639a2bcd4740e4be7b2b162004669a because the tablet was not found. No further retry: Not found: Tablet not found: 1c639a2bcd4740e4be7b2b162004669a
I20250112 02:52:39.506825 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 6117dc8b32a64535a2f4167a03bed0d2 (table default.Table_365739770a11434dbf260ea912e7ea81 [id=868adb627654450c936ef198e49d853b]). Expected version 1 got 0
I20250112 02:52:39.507752 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:39.510371 2562 ts_tablet_manager.cc:1939] T b1d9fae290ec4281839991f9d40b105c P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:39.510989 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:39.514026 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet b1d9fae290ec4281839991f9d40b105c (table default.Table_c1639450ddf94784a40b8cf28f3059c4 [id=93cb911ed4e045c994c9a06b10e2e4a8]) successfully deleted
W20250112 02:52:39.514891 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 06edfca60eed4dd3a2dee7b662b0970c because the tablet was not found. No further retry: Not found: Tablet not found: 06edfca60eed4dd3a2dee7b662b0970c
I20250112 02:52:39.516851 2003 raft_consensus.cc:2267] T 87759e8b99b04e628badc8f60cb38bb3 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.517136 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet fccff64af8ca4400afc116db98918119 (table default.Table_15bcb2c0f2a74458a1bed231fd3437fc [id=ca9c8281bf254f248076cb600e7c4f87]). Expected version 2 got 0
I20250112 02:52:39.517520 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:39.517776 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:39.517913 2003 ts_tablet_manager.cc:1905] T 87759e8b99b04e628badc8f60cb38bb3 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:39.518062 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet f7cd0927c6a4401cbe69f55d4db9963a because the tablet was not found. No further retry: Not found: Tablet not found: f7cd0927c6a4401cbe69f55d4db9963a
W20250112 02:52:39.518332 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet d659c5a4f7a446c6aaf2283f65b45106 because the tablet was not found. No further retry: Not found: Tablet not found: d659c5a4f7a446c6aaf2283f65b45106
W20250112 02:52:39.518579 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 02d0f67141dc46b6a87ca0ad080d5bfe because the tablet was not found. No further retry: Not found: Tablet not found: 02d0f67141dc46b6a87ca0ad080d5bfe
I20250112 02:52:39.518800 2562 tablet_replica.cc:331] T d63d411f906b4d4eb68343bdcc493e05 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
W20250112 02:52:39.518818 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet cc62dcbf67e84e36ae1fbe127ccd4879 because the tablet was not found. No further retry: Not found: Tablet not found: cc62dcbf67e84e36ae1fbe127ccd4879
I20250112 02:52:39.518993 2562 raft_consensus.cc:2238] T d63d411f906b4d4eb68343bdcc493e05 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:39.519117 2562 raft_consensus.cc:2267] T d63d411f906b4d4eb68343bdcc493e05 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.519274 1181 tablet_bootstrap.cc:654] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.519438 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:39.519631 2562 ts_tablet_manager.cc:1905] T d63d411f906b4d4eb68343bdcc493e05 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.527523 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
I20250112 02:52:39.527796 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:39.528304 455 catalog_manager.cc:5526] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } }
W20250112 02:52:39.528946 989 consensus_peers.cc:487] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f05c8426ea3d4fe89fbe6c88c688019f. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:39.529217 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:39.531348 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5cc414d6420c41839621ab858067548b (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]). Expected version 1 got 0
I20250112 02:52:39.534993 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:39.535638 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:39.536412 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:39.536662 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:39.536939 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:39.537130 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:39.537560 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:39.538100 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:39.538357 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
I20250112 02:52:39.538552 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:39.540095 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:39.540844 2562 ts_tablet_manager.cc:1918] T d63d411f906b4d4eb68343bdcc493e05 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:39.541005 2562 log.cc:1198] T d63d411f906b4d4eb68343bdcc493e05 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/d63d411f906b4d4eb68343bdcc493e05
I20250112 02:52:39.541316 2562 ts_tablet_manager.cc:1939] T d63d411f906b4d4eb68343bdcc493e05 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:39.543244 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet d63d411f906b4d4eb68343bdcc493e05 (table default.Table_e80769df10444dd5ae4bc23a8c6cc09e [id=d91a9f9e1f76482e99d4f10c0568f2e3]) successfully deleted
W20250112 02:52:39.544212 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet cbe30f9860e2492b90e8cedf8ff9daba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbe30f9860e2492b90e8cedf8ff9daba already in progress: creating tablet
I20250112 02:52:39.516443 1177 tablet_bootstrap.cc:492] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:39.547716 1177 ts_tablet_manager.cc:1397] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.000s
I20250112 02:52:39.548545 2562 tablet_replica.cc:331] T a3ffdc237f8640f0b378809f93fc6292 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
W20250112 02:52:39.548540 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet daef3a58605e40978f7389698b22265d because the tablet was not found. No further retry: Not found: Tablet not found: daef3a58605e40978f7389698b22265d
I20250112 02:52:39.548537 1177 raft_consensus.cc:357] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.548777 1177 raft_consensus.cc:383] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.548952 1177 raft_consensus.cc:738] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.549278 1177 consensus_queue.cc:260] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.550141 2562 raft_consensus.cc:2238] T a3ffdc237f8640f0b378809f93fc6292 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:39.550267 1177 ts_tablet_manager.cc:1428] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:39.550581 1177 tablet_bootstrap.cc:492] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:39.551355 2562 raft_consensus.cc:2267] T a3ffdc237f8640f0b378809f93fc6292 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.552094 2562 ts_tablet_manager.cc:1905] T a3ffdc237f8640f0b378809f93fc6292 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.563853 1177 tablet_bootstrap.cc:654] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.576918 2003 ts_tablet_manager.cc:1918] T 87759e8b99b04e628badc8f60cb38bb3 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:39.584021 2228 raft_consensus.cc:491] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.584161 2228 raft_consensus.cc:513] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.584725 2228 leader_election.cc:290] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.585979 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfd1ebe729d458ea31eb7feff7ee072" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:39.586154 910 raft_consensus.cc:2463] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:39.586567 989 leader_election.cc:304] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:39.587082 2228 raft_consensus.cc:2798] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:39.587157 2228 raft_consensus.cc:491] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:39.587217 2228 raft_consensus.cc:3054] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.589536 2228 raft_consensus.cc:513] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.590235 1181 tablet_bootstrap.cc:492] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:39.590394 1181 ts_tablet_manager.cc:1397] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.113s user 0.008s sys 0.000s
I20250112 02:52:39.591006 1181 raft_consensus.cc:357] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:39.591208 1181 raft_consensus.cc:383] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.591269 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfd1ebe729d458ea31eb7feff7ee072" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:39.591408 910 raft_consensus.cc:3054] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.593765 910 raft_consensus.cc:2463] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:39.594385 989 leader_election.cc:304] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:39.594864 2326 raft_consensus.cc:2798] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:39.594964 2326 raft_consensus.cc:695] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:39.595217 2326 consensus_queue.cc:237] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.597335 2228 leader_election.cc:290] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.598134 2024 ts_tablet_manager.cc:1918] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:39.591284 1181 raft_consensus.cc:738] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.598291 2024 log.cc:1198] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/2fdeaf5fa8cc4a2b97b2178d8ecbad69
I20250112 02:52:39.598778 2024 ts_tablet_manager.cc:1939] T 2fdeaf5fa8cc4a2b97b2178d8ecbad69 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:39.599211 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 78) took 5066 ms. Trace:
I20250112 02:52:39.599350 2024 rpcz_store.cc:276] 0112 02:52:34.532633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.532705 (+ 72us) service_pool.cc:225] Handling call
0112 02:52:39.457961 (+4925256us) ts_tablet_manager.cc:1145] Deleting tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69
0112 02:52:39.457970 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.502278 (+ 44308us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.598039 (+ 95761us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.599181 (+ 1142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4915197}
I20250112 02:52:39.598949 1181 consensus_queue.cc:260] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:39.585587 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfd1ebe729d458ea31eb7feff7ee072" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:39.599999 2003 log.cc:1198] T 87759e8b99b04e628badc8f60cb38bb3 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/87759e8b99b04e628badc8f60cb38bb3
I20250112 02:52:39.600061 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]) successfully deleted
I20250112 02:52:39.600365 2003 ts_tablet_manager.cc:1939] T 87759e8b99b04e628badc8f60cb38bb3 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
W20250112 02:52:39.600365 991 leader_election.cc:343] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.600414 2024 tablet_replica.cc:331] T 8a12b225131b464494a9f02133d37aea P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:39.600701 2024 raft_consensus.cc:2238] T 8a12b225131b464494a9f02133d37aea P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:39.586781 1177 tablet_bootstrap.cc:492] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:39.590585 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfd1ebe729d458ea31eb7feff7ee072" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:39.599380 2451 consensus_queue.cc:1035] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:39.602223 1177 ts_tablet_manager.cc:1397] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.000s
I20250112 02:52:39.603046 1177 raft_consensus.cc:357] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.603305 1177 raft_consensus.cc:383] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.603410 1177 raft_consensus.cc:738] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.603646 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 87759e8b99b04e628badc8f60cb38bb3 (table default.Table_abf2c7b9b8da40c6b9a8381b4e41bda6 [id=4a81fc17cbb94a23a53b5430b545a756]) successfully deleted
I20250112 02:52:39.603668 1177 consensus_queue.cc:260] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.606424 747 tablet_service.cc:1467] Processing CreateTablet for tablet ca2165362ee541adac724b63f39b1489 (DEFAULT_TABLE table=default.Table_5fb592e41cc4406aa27546e7b41fdb88 [id=6402abce8d1d45c182cc989490c6a5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:39.606760 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca2165362ee541adac724b63f39b1489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:39.607810 1181 ts_tablet_manager.cc:1428] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250112 02:52:39.608631 1181 tablet_bootstrap.cc:492] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:39.610780 895 tablet_service.cc:1467] Processing CreateTablet for tablet ca2165362ee541adac724b63f39b1489 (DEFAULT_TABLE table=default.Table_5fb592e41cc4406aa27546e7b41fdb88 [id=6402abce8d1d45c182cc989490c6a5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:39.611075 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca2165362ee541adac724b63f39b1489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:39.618448 1177 ts_tablet_manager.cc:1428] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
W20250112 02:52:39.601706 991 leader_election.cc:343] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.609335 1031 tablet_service.cc:1467] Processing CreateTablet for tablet ca2165362ee541adac724b63f39b1489 (DEFAULT_TABLE table=default.Table_5fb592e41cc4406aa27546e7b41fdb88 [id=6402abce8d1d45c182cc989490c6a5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:39.618088 1173 tablet_bootstrap.cc:492] T afd6d31155034908ac44a90c3a2ea7d6 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:39.621874 1173 ts_tablet_manager.cc:1397] T afd6d31155034908ac44a90c3a2ea7d6 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.139s user 0.003s sys 0.004s
I20250112 02:52:39.622002 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca2165362ee541adac724b63f39b1489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:39.622506 1173 raft_consensus.cc:357] T afd6d31155034908ac44a90c3a2ea7d6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.622678 1173 raft_consensus.cc:383] T afd6d31155034908ac44a90c3a2ea7d6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.622720 1173 raft_consensus.cc:738] T afd6d31155034908ac44a90c3a2ea7d6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.622923 1173 consensus_queue.cc:260] T afd6d31155034908ac44a90c3a2ea7d6 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.623286 1173 ts_tablet_manager.cc:1428] T afd6d31155034908ac44a90c3a2ea7d6 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250112 02:52:39.623492 1173 tablet_bootstrap.cc:492] T ead501d76bb1412e9839fb36b9911e6f P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:39.624950 1173 tablet_bootstrap.cc:654] T ead501d76bb1412e9839fb36b9911e6f P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:39.626070 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 70715db0394d46d9b6b289160b00be7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70715db0394d46d9b6b289160b00be7d already in progress: creating tablet
W20250112 02:52:39.626350 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 54215c55966a46e1ab96e352c4e3764e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54215c55966a46e1ab96e352c4e3764e already in progress: creating tablet
I20250112 02:52:39.627102 2003 tablet_replica.cc:331] T f05c8426ea3d4fe89fbe6c88c688019f P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:39.627267 2003 raft_consensus.cc:2238] T f05c8426ea3d4fe89fbe6c88c688019f P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250112 02:52:39.627246 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 65f22a7b1b804661af3b7e785a833739 because the tablet was not found. No further retry: Not found: Tablet not found: 65f22a7b1b804661af3b7e785a833739
I20250112 02:52:39.627419 2003 raft_consensus.cc:2267] T f05c8426ea3d4fe89fbe6c88c688019f P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:39.627453 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 49bb6d20cfa0453e90e048825114525e because the tablet was not found. No further retry: Not found: Tablet not found: 49bb6d20cfa0453e90e048825114525e
W20250112 02:52:39.627666 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b9199b0d2b924a79938a979df4196bb4 because the tablet was not found. No further retry: Not found: Tablet not found: b9199b0d2b924a79938a979df4196bb4
W20250112 02:52:39.627848 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b9c006281dc949119951c74e23167d7f because the tablet was not found. No further retry: Not found: Tablet not found: b9c006281dc949119951c74e23167d7f
I20250112 02:52:39.628648 1181 tablet_bootstrap.cc:654] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.631347 2562 ts_tablet_manager.cc:1918] T a3ffdc237f8640f0b378809f93fc6292 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:39.631459 2562 log.cc:1198] T a3ffdc237f8640f0b378809f93fc6292 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/a3ffdc237f8640f0b378809f93fc6292
I20250112 02:52:39.631800 2562 ts_tablet_manager.cc:1939] T a3ffdc237f8640f0b378809f93fc6292 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:39.632707 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet a3ffdc237f8640f0b378809f93fc6292 (table default.Table_fc264b6a06584c7ea757f5d18e05e2bc [id=b560a3488e3d4db58abc7d81d6094d86]) successfully deleted
I20250112 02:52:39.634230 2451 consensus_queue.cc:1035] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:39.634124 2024 raft_consensus.cc:2267] T 8a12b225131b464494a9f02133d37aea P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.641069 2024 ts_tablet_manager.cc:1905] T 8a12b225131b464494a9f02133d37aea P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.644517 2003 ts_tablet_manager.cc:1905] T f05c8426ea3d4fe89fbe6c88c688019f P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.644745 2598 raft_consensus.cc:491] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.644879 2598 raft_consensus.cc:513] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.645392 2598 leader_election.cc:290] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:39.645991 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd81852962fb43449e35ef45dc1daaad" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:39.646754 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd81852962fb43449e35ef45dc1daaad" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:39.647287 855 leader_election.cc:343] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.647686 2562 tablet_replica.cc:331] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
W20250112 02:52:39.647665 854 leader_election.cc:343] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.647790 854 leader_election.cc:304] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:39.647888 2562 raft_consensus.cc:2238] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:39.648576 2609 raft_consensus.cc:491] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.648717 2609 raft_consensus.cc:513] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.649505 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d1beac4410422bbf1e912d35999f3d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:39.649659 2609 leader_election.cc:290] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:39.650477 991 leader_election.cc:343] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.648005 2562 raft_consensus.cc:2267] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.648226 2598 raft_consensus.cc:2743] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:39.650115 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d1beac4410422bbf1e912d35999f3d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:39.653223 917 raft_consensus.cc:2463] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:39.653632 989 leader_election.cc:304] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:39.656069 2609 raft_consensus.cc:2798] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:39.656200 2609 raft_consensus.cc:491] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:39.656313 2609 raft_consensus.cc:3054] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.656322 1173 tablet_bootstrap.cc:492] T ead501d76bb1412e9839fb36b9911e6f P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:39.658308 1173 ts_tablet_manager.cc:1397] T ead501d76bb1412e9839fb36b9911e6f P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20250112 02:52:39.658857 2609 raft_consensus.cc:513] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.658949 1173 raft_consensus.cc:357] T ead501d76bb1412e9839fb36b9911e6f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.659106 1173 raft_consensus.cc:383] T ead501d76bb1412e9839fb36b9911e6f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.659168 1173 raft_consensus.cc:738] T ead501d76bb1412e9839fb36b9911e6f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.659368 1173 consensus_queue.cc:260] T ead501d76bb1412e9839fb36b9911e6f P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.659821 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d1beac4410422bbf1e912d35999f3d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:39.659967 1173 ts_tablet_manager.cc:1428] T ead501d76bb1412e9839fb36b9911e6f P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:39.660153 1181 tablet_bootstrap.cc:492] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:39.660195 1173 tablet_bootstrap.cc:492] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:39.660285 991 leader_election.cc:343] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.660630 1181 ts_tablet_manager.cc:1397] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
I20250112 02:52:39.661078 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d1beac4410422bbf1e912d35999f3d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:39.661268 910 raft_consensus.cc:3054] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.663691 910 raft_consensus.cc:2463] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:39.664090 989 leader_election.cc:304] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:39.664280 2326 raft_consensus.cc:2798] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:39.664371 2609 leader_election.cc:290] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.664481 2326 raft_consensus.cc:695] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:39.664710 2326 consensus_queue.cc:237] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.664328 1181 raft_consensus.cc:357] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.665299 1181 raft_consensus.cc:383] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.665570 1181 raft_consensus.cc:738] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.665953 1173 tablet_bootstrap.cc:654] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.666715 1181 consensus_queue.cc:260] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.671082 1181 ts_tablet_manager.cc:1428] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250112 02:52:39.672152 1181 tablet_bootstrap.cc:492] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:39.672508 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:39.677279 2326 raft_consensus.cc:491] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.677469 2326 raft_consensus.cc:513] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.678463 2003 ts_tablet_manager.cc:1918] T f05c8426ea3d4fe89fbe6c88c688019f P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:39.678542 2326 leader_election.cc:290] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.678790 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "781fe3514f4c4f80b1ee49278502ccc1" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:39.679028 910 raft_consensus.cc:2463] T 781fe3514f4c4f80b1ee49278502ccc1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:39.679366 989 leader_election.cc:304] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:39.679721 2228 raft_consensus.cc:2798] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:39.679793 2228 raft_consensus.cc:491] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:39.679862 2228 raft_consensus.cc:3054] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.682127 2228 raft_consensus.cc:513] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.682606 2228 leader_election.cc:290] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.683655 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "781fe3514f4c4f80b1ee49278502ccc1" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:39.683801 917 raft_consensus.cc:3054] T 781fe3514f4c4f80b1ee49278502ccc1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.678562 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "781fe3514f4c4f80b1ee49278502ccc1" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:39.684093 779 raft_consensus.cc:2463] T 781fe3514f4c4f80b1ee49278502ccc1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:39.683836 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "781fe3514f4c4f80b1ee49278502ccc1" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:39.684257 774 raft_consensus.cc:3054] T 781fe3514f4c4f80b1ee49278502ccc1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.685058 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:39.686110 917 raft_consensus.cc:2463] T 781fe3514f4c4f80b1ee49278502ccc1 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:39.665059 2562 ts_tablet_manager.cc:1905] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.686477 774 raft_consensus.cc:2463] T 781fe3514f4c4f80b1ee49278502ccc1 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:39.686787 989 leader_election.cc:304] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:39.687069 2228 raft_consensus.cc:2798] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:39.687186 2228 raft_consensus.cc:695] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:39.687285 2003 log.cc:1198] T f05c8426ea3d4fe89fbe6c88c688019f P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/f05c8426ea3d4fe89fbe6c88c688019f
I20250112 02:52:39.687729 2003 ts_tablet_manager.cc:1939] T f05c8426ea3d4fe89fbe6c88c688019f P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:39.687536 2228 consensus_queue.cc:237] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.688268 2024 ts_tablet_manager.cc:1918] T 8a12b225131b464494a9f02133d37aea P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:39.689313 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet f05c8426ea3d4fe89fbe6c88c688019f (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]) successfully deleted
I20250112 02:52:39.689556 2024 log.cc:1198] T 8a12b225131b464494a9f02133d37aea P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/8a12b225131b464494a9f02133d37aea
I20250112 02:52:39.690115 1173 tablet_bootstrap.cc:492] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:39.690225 1173 ts_tablet_manager.cc:1397] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20250112 02:52:39.690865 1173 raft_consensus.cc:357] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.691037 1173 raft_consensus.cc:383] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.691092 1173 raft_consensus.cc:738] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.691263 1173 consensus_queue.cc:260] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.691816 1173 ts_tablet_manager.cc:1428] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:39.692092 1173 tablet_bootstrap.cc:492] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:39.692643 1181 tablet_bootstrap.cc:654] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.693632 1173 tablet_bootstrap.cc:654] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.693404 2024 ts_tablet_manager.cc:1939] T 8a12b225131b464494a9f02133d37aea P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:39.694286 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:39.696122 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 79) took 5163 ms. Trace:
I20250112 02:52:39.696273 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:39.696794 2024 rpcz_store.cc:276] 0112 02:52:34.532760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.532810 (+ 50us) service_pool.cc:225] Handling call
0112 02:52:39.600342 (+5067532us) ts_tablet_manager.cc:1145] Deleting tablet 8a12b225131b464494a9f02133d37aea
0112 02:52:39.600378 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.660473 (+ 60095us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.687425 (+ 26952us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.695831 (+ 8406us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5057540}
I20250112 02:52:39.698391 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 8a12b225131b464494a9f02133d37aea (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]) successfully deleted
I20250112 02:52:39.698611 827 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:39.699277 2024 tablet_replica.cc:331] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:39.699858 2445 raft_consensus.cc:491] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.699986 2445 raft_consensus.cc:513] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.700491 2445 leader_election.cc:290] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.700189 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_5d473cabff2e46e0b531fbce28d9d558 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 1 got 0
I20250112 02:52:39.700548 2024 raft_consensus.cc:2238] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:39.701365 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead501d76bb1412e9839fb36b9911e6f" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:39.701746 910 raft_consensus.cc:2463] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:39.701902 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead501d76bb1412e9839fb36b9911e6f" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:39.701622 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:39.702404 713 leader_election.cc:304] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:39.702694 2445 raft_consensus.cc:2798] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:39.702771 2445 raft_consensus.cc:491] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:39.702836 2445 raft_consensus.cc:3054] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.703069 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:39.703990 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_5e7392cb5a68412f97bddb2677986918 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 1 got 0
I20250112 02:52:39.705199 2445 raft_consensus.cc:513] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.706561 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead501d76bb1412e9839fb36b9911e6f" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:39.706719 917 raft_consensus.cc:3054] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.709226 917 raft_consensus.cc:2463] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:39.709560 713 leader_election.cc:304] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:39.709769 2179 raft_consensus.cc:2798] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:39.709897 2179 raft_consensus.cc:695] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:39.710083 2179 consensus_queue.cc:237] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.710378 1181 tablet_bootstrap.cc:492] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:39.710538 1181 ts_tablet_manager.cc:1397] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20250112 02:52:39.711395 2445 leader_election.cc:290] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.712018 1181 raft_consensus.cc:357] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.713027 1181 raft_consensus.cc:383] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.713151 1181 raft_consensus.cc:738] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.713817 1181 consensus_queue.cc:260] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.717129 2003 tablet_replica.cc:331] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:39.719632 2003 raft_consensus.cc:2238] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:39.719817 2003 raft_consensus.cc:2267] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.719830 1181 ts_tablet_manager.cc:1428] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250112 02:52:39.720559 2003 ts_tablet_manager.cc:1905] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.721047 1181 tablet_bootstrap.cc:492] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:39.723011 2598 raft_consensus.cc:491] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.723119 2598 raft_consensus.cc:513] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.723662 2598 leader_election.cc:290] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:39.724685 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b124c15736084e6d8f69b4fb7c41cf39" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:39.725126 855 leader_election.cc:343] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:39.725580 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 2730634a5cfd4c2fb5c4e30100c556d4 because the tablet was not found. No further retry: Not found: Tablet not found: 2730634a5cfd4c2fb5c4e30100c556d4
W20250112 02:52:39.725905 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 06edfca60eed4dd3a2dee7b662b0970c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06edfca60eed4dd3a2dee7b662b0970c already in progress: creating tablet
W20250112 02:52:39.726213 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b61f16f3d601436bb4b32344de28c1a7 because the tablet was not found. No further retry: Not found: Tablet not found: b61f16f3d601436bb4b32344de28c1a7
W20250112 02:52:39.726390 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 3fd498ccce484a89855c982762086890 because the tablet was not found. No further retry: Not found: Tablet not found: 3fd498ccce484a89855c982762086890
W20250112 02:52:39.726557 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet cc62dcbf67e84e36ae1fbe127ccd4879 because the tablet was not found. No further retry: Not found: Tablet not found: cc62dcbf67e84e36ae1fbe127ccd4879
W20250112 02:52:39.726715 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet bab4978edea0445fb0456e88e7fb4c44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bab4978edea0445fb0456e88e7fb4c44 already in progress: creating tablet
W20250112 02:52:39.726897 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 1a1e3af2883341e9b957f6ed748e604d because the tablet was not found. No further retry: Not found: Tablet not found: 1a1e3af2883341e9b957f6ed748e604d
W20250112 02:52:39.727056 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet c7c530aa0aa64e47ae27d61f341d2bd9 because the tablet was not found. No further retry: Not found: Tablet not found: c7c530aa0aa64e47ae27d61f341d2bd9
I20250112 02:52:39.727210 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 1 got 0
I20250112 02:52:39.727517 446 catalog_manager.cc:5526] T f0dc78d722ac4687bfb3dce94d6db7f0 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } }
I20250112 02:52:39.728888 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:39.731814 2179 raft_consensus.cc:491] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.731940 2179 raft_consensus.cc:513] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.732937 2179 leader_election.cc:290] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.733150 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc8658b0efe43bdb339edbff64973f6" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:39.733318 917 raft_consensus.cc:2463] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:39.733639 713 leader_election.cc:304] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:39.733975 2179 raft_consensus.cc:2798] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:39.734047 2179 raft_consensus.cc:491] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:39.734107 2179 raft_consensus.cc:3054] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.734678 2003 ts_tablet_manager.cc:1918] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:39.734767 2003 log.cc:1198] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/fe614eaeb793473d9b2da206b7345562
I20250112 02:52:39.735070 2003 ts_tablet_manager.cc:1939] T fe614eaeb793473d9b2da206b7345562 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:39.702551 2024 raft_consensus.cc:2267] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.705991 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead501d76bb1412e9839fb36b9911e6f" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:39.725028 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b124c15736084e6d8f69b4fb7c41cf39" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:39.725996 1173 tablet_bootstrap.cc:492] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:39.732791 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc8658b0efe43bdb339edbff64973f6" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:39.734236 2215 raft_consensus.cc:491] T ead501d76bb1412e9839fb36b9911e6f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.736188 2215 raft_consensus.cc:513] T ead501d76bb1412e9839fb36b9911e6f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:39.736397 854 leader_election.cc:343] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.736511 854 leader_election.cc:304] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:39.736761 2215 leader_election.cc:290] T ead501d76bb1412e9839fb36b9911e6f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.736855 2598 raft_consensus.cc:2743] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:39.736653 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:39.737507 1053 raft_consensus.cc:3054] T ead501d76bb1412e9839fb36b9911e6f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.738018 2024 ts_tablet_manager.cc:1905] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.739913 1053 raft_consensus.cc:2463] T ead501d76bb1412e9839fb36b9911e6f P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:39.741219 1173 ts_tablet_manager.cc:1397] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20250112 02:52:39.741972 1173 raft_consensus.cc:357] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.742190 1173 raft_consensus.cc:383] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.742288 1173 raft_consensus.cc:738] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.742528 1173 consensus_queue.cc:260] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.743161 1173 ts_tablet_manager.cc:1428] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:39.743217 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead501d76bb1412e9839fb36b9911e6f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:39.743418 1173 tablet_bootstrap.cc:492] T 4fc8658b0efe43bdb339edbff64973f6 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:39.743494 917 raft_consensus.cc:2388] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1e775296eb746db8b318f17d26820e6 in current term 1: Already voted for candidate 289afc116aa74408aa5bff74e11c50d8 in this term.
I20250112 02:52:39.744830 1050 raft_consensus.cc:2371] T ead501d76bb1412e9839fb36b9911e6f P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1. Re-sending same reply.
I20250112 02:52:39.745267 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:39.746073 1181 tablet_bootstrap.cc:654] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.746630 2598 raft_consensus.cc:491] T 9ded8bcdda914ea6bc4f9bb92d413dee P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.746842 2598 raft_consensus.cc:513] T 9ded8bcdda914ea6bc4f9bb92d413dee P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.747393 2598 leader_election.cc:290] T 9ded8bcdda914ea6bc4f9bb92d413dee P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:39.748169 2003 tablet_replica.cc:331] T c4a3486ef011492c8a1df8d6278adedf P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:39.748198 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ded8bcdda914ea6bc4f9bb92d413dee" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:39.748205 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ded8bcdda914ea6bc4f9bb92d413dee" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:39.748329 2003 raft_consensus.cc:2238] T c4a3486ef011492c8a1df8d6278adedf P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:39.748454 2003 raft_consensus.cc:2267] T c4a3486ef011492c8a1df8d6278adedf P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:39.748667 855 leader_election.cc:343] T 9ded8bcdda914ea6bc4f9bb92d413dee P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:39.749035 854 leader_election.cc:343] T 9ded8bcdda914ea6bc4f9bb92d413dee P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.749118 854 leader_election.cc:304] T 9ded8bcdda914ea6bc4f9bb92d413dee P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:39.749434 2598 raft_consensus.cc:2743] T 9ded8bcdda914ea6bc4f9bb92d413dee P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:39.751207 2562 ts_tablet_manager.cc:1918] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:39.751789 2562 log.cc:1198] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/54215c55966a46e1ab96e352c4e3764e
I20250112 02:52:39.754971 2562 ts_tablet_manager.cc:1939] T 54215c55966a46e1ab96e352c4e3764e P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:39.756069 2215 raft_consensus.cc:491] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.756170 2215 raft_consensus.cc:513] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:39.755965 446 catalog_manager.cc:5526] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } }
I20250112 02:52:39.756652 2215 leader_election.cc:290] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.757088 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f924459d1150472aafb325a11825db23" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:39.737238 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead501d76bb1412e9839fb36b9911e6f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:39.757233 910 raft_consensus.cc:2463] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
W20250112 02:52:39.740888 714 leader_election.cc:343] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.757580 989 leader_election.cc:304] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:39.757995 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet fe614eaeb793473d9b2da206b7345562 (table default.Table_b2b3c738a24f451eac39fb1a6399f37a [id=7c6c2713869e476cb23d157a285a5afd]) successfully deleted
W20250112 02:52:39.758220 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 41e166b3f8414ababbd4a3f07fb90c4c because the tablet was not found. No further retry: Not found: Tablet not found: 41e166b3f8414ababbd4a3f07fb90c4c
W20250112 02:52:39.758437 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 015412a5fa304453bd8d1be1c3cd7a46 because the tablet was not found. No further retry: Not found: Tablet not found: 015412a5fa304453bd8d1be1c3cd7a46
I20250112 02:52:39.758456 991 leader_election.cc:304] T ead501d76bb1412e9839fb36b9911e6f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
W20250112 02:52:39.758607 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 7f4e4f1e451b48dc9cdd780300f6888c because the tablet was not found. No further retry: Not found: Tablet not found: 7f4e4f1e451b48dc9cdd780300f6888c
I20250112 02:52:39.758759 2215 raft_consensus.cc:2798] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:39.758824 2215 raft_consensus.cc:491] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250112 02:52:39.758818 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 8a12b225131b464494a9f02133d37aea because the tablet was not found. No further retry: Not found: Tablet not found: 8a12b225131b464494a9f02133d37aea
I20250112 02:52:39.758880 2215 raft_consensus.cc:3054] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:39.759022 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet ac716c6da01941a989754aac1ef430d5 because the tablet was not found. No further retry: Not found: Tablet not found: ac716c6da01941a989754aac1ef430d5
I20250112 02:52:39.759049 2228 raft_consensus.cc:2743] T ead501d76bb1412e9839fb36b9911e6f P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:39.759202 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 61593969d2234ce3a6089027b067d922 because the tablet was not found. No further retry: Not found: Tablet not found: 61593969d2234ce3a6089027b067d922
I20250112 02:52:39.745322 1177 tablet_bootstrap.cc:492] T bb4ef0220dde43569452379639759db5 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:39.759351 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 3c686fdc8896463c81c9f7c5f53a3da2 because the tablet was not found. No further retry: Not found: Tablet not found: 3c686fdc8896463c81c9f7c5f53a3da2
I20250112 02:52:39.736285 2179 raft_consensus.cc:513] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.757141 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f924459d1150472aafb325a11825db23" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:39.759521 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 1a8a178bafcb457f9b0e9587df560493 because the tablet was not found. No further retry: Not found: Tablet not found: 1a8a178bafcb457f9b0e9587df560493
I20250112 02:52:39.759588 779 raft_consensus.cc:2463] T f924459d1150472aafb325a11825db23 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
W20250112 02:52:39.759688 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 64707ee432fb4e96a3bbf7252dfa1813 because the tablet was not found. No further retry: Not found: Tablet not found: 64707ee432fb4e96a3bbf7252dfa1813
W20250112 02:52:39.759904 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 42cb7e7a41634cf1bdb978ac9d2befcd because the tablet was not found. No further retry: Not found: Tablet not found: 42cb7e7a41634cf1bdb978ac9d2befcd
W20250112 02:52:39.760095 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 35f59617b34046568dd83bbb7cdc42d2 because the tablet was not found. No further retry: Not found: Tablet not found: 35f59617b34046568dd83bbb7cdc42d2
I20250112 02:52:39.760643 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc8658b0efe43bdb339edbff64973f6" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:39.761057 714 leader_election.cc:343] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.761972 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:39.762843 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:39.763317 2179 leader_election.cc:290] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.763381 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc8658b0efe43bdb339edbff64973f6" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:39.763535 910 raft_consensus.cc:3054] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.765050 2215 raft_consensus.cc:513] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:39.765621 1177 tablet_bootstrap.cc:654] T bb4ef0220dde43569452379639759db5 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.765815 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f924459d1150472aafb325a11825db23" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:39.765962 779 raft_consensus.cc:3054] T f924459d1150472aafb325a11825db23 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.766003 910 raft_consensus.cc:2463] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:39.766319 713 leader_election.cc:304] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:39.766629 2003 ts_tablet_manager.cc:1905] T c4a3486ef011492c8a1df8d6278adedf P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.766670 2179 raft_consensus.cc:2798] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:39.766872 2179 raft_consensus.cc:695] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:39.767066 2179 consensus_queue.cc:237] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.768391 779 raft_consensus.cc:2463] T f924459d1150472aafb325a11825db23 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:39.768922 991 leader_election.cc:304] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:39.769285 2215 leader_election.cc:290] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.769467 2215 raft_consensus.cc:2798] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:39.769593 2215 raft_consensus.cc:695] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:39.769647 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f924459d1150472aafb325a11825db23" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:39.769832 910 raft_consensus.cc:3054] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.769820 2215 consensus_queue.cc:237] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:39.770677 1173 tablet_bootstrap.cc:654] T 4fc8658b0efe43bdb339edbff64973f6 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.772075 910 raft_consensus.cc:2463] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:39.774134 2024 ts_tablet_manager.cc:1918] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:39.774267 2024 log.cc:1198] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/f05c8426ea3d4fe89fbe6c88c688019f
I20250112 02:52:39.774650 2024 ts_tablet_manager.cc:1939] T f05c8426ea3d4fe89fbe6c88c688019f P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:39.774205 2228 consensus_queue.cc:1035] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:39.775940 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 81) took 5242 ms. Trace:
I20250112 02:52:39.776109 2024 rpcz_store.cc:276] 0112 02:52:34.533458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.533494 (+ 36us) service_pool.cc:225] Handling call
0112 02:52:39.699030 (+5165536us) ts_tablet_manager.cc:1145] Deleting tablet f05c8426ea3d4fe89fbe6c88c688019f
0112 02:52:39.699038 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.740602 (+ 41564us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.774031 (+ 33429us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.775908 (+ 1877us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5156176}
I20250112 02:52:39.776959 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet f05c8426ea3d4fe89fbe6c88c688019f (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]) successfully deleted
I20250112 02:52:39.777077 2024 tablet_replica.cc:331] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:39.777272 2024 raft_consensus.cc:2238] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:39.777560 2024 raft_consensus.cc:2267] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.779642 2024 ts_tablet_manager.cc:1905] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.780772 2003 ts_tablet_manager.cc:1918] T c4a3486ef011492c8a1df8d6278adedf P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:39.774677 446 catalog_manager.cc:5526] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } }
I20250112 02:52:39.785504 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:39.786466 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
I20250112 02:52:39.787772 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:39.789669 2003 log.cc:1198] T c4a3486ef011492c8a1df8d6278adedf P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/c4a3486ef011492c8a1df8d6278adedf
I20250112 02:52:39.789999 2003 ts_tablet_manager.cc:1939] T c4a3486ef011492c8a1df8d6278adedf P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:39.789824 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:39.791005 1177 tablet_bootstrap.cc:492] T bb4ef0220dde43569452379639759db5 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:39.791150 1177 ts_tablet_manager.cc:1397] T bb4ef0220dde43569452379639759db5 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.000s
I20250112 02:52:39.791764 1177 raft_consensus.cc:357] T bb4ef0220dde43569452379639759db5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.792008 1177 raft_consensus.cc:383] T bb4ef0220dde43569452379639759db5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.792112 1177 raft_consensus.cc:738] T bb4ef0220dde43569452379639759db5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.792364 1177 consensus_queue.cc:260] T bb4ef0220dde43569452379639759db5 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.794000 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:39.794354 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:39.794740 446 catalog_manager.cc:5526] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } }
I20250112 02:52:39.795065 2609 consensus_queue.cc:1035] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250112 02:52:39.795688 1177 ts_tablet_manager.cc:1428] T bb4ef0220dde43569452379639759db5 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250112 02:52:39.795985 1177 tablet_bootstrap.cc:492] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:39.797158 1177 tablet_bootstrap.cc:654] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.797179 1173 tablet_bootstrap.cc:492] T 4fc8658b0efe43bdb339edbff64973f6 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:39.797880 1173 ts_tablet_manager.cc:1397] T 4fc8658b0efe43bdb339edbff64973f6 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.001s
I20250112 02:52:39.798568 1173 raft_consensus.cc:357] T 4fc8658b0efe43bdb339edbff64973f6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.799197 1173 raft_consensus.cc:383] T 4fc8658b0efe43bdb339edbff64973f6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.799196 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet c4a3486ef011492c8a1df8d6278adedf (table default.Table_33b6530b130041f0a2a4a7240e92ff15 [id=1d614fd3947a43bcac5a3d40cad6d80b]) successfully deleted
I20250112 02:52:39.799458 1173 raft_consensus.cc:738] T 4fc8658b0efe43bdb339edbff64973f6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.799933 1173 consensus_queue.cc:260] T 4fc8658b0efe43bdb339edbff64973f6 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.800711 1173 ts_tablet_manager.cc:1428] T 4fc8658b0efe43bdb339edbff64973f6 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 02:52:39.801524 1173 tablet_bootstrap.cc:492] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:39.801527 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b7bd96fb381f4e73a0081be3fc799c36 because the tablet was not found. No further retry: Not found: Tablet not found: b7bd96fb381f4e73a0081be3fc799c36
W20250112 02:52:39.801807 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet a12c5d58e6c246c983e8a533906ce77d because the tablet was not found. No further retry: Not found: Tablet not found: a12c5d58e6c246c983e8a533906ce77d
I20250112 02:52:39.804679 2003 tablet_replica.cc:331] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:39.804898 2003 raft_consensus.cc:2238] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
W20250112 02:52:39.805169 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet fb0aba0139cf479b80b32f3dfc936535 because the tablet was not found. No further retry: Not found: Tablet not found: fb0aba0139cf479b80b32f3dfc936535
I20250112 02:52:39.805210 2003 raft_consensus.cc:2267] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.806011 2003 ts_tablet_manager.cc:1905] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.813395 1177 tablet_bootstrap.cc:492] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:39.813541 1177 ts_tablet_manager.cc:1397] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20250112 02:52:39.814088 1177 raft_consensus.cc:357] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.815310 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 162) took 1002 ms. Trace:
I20250112 02:52:39.815441 2562 rpcz_store.cc:276] 0112 02:52:38.812967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.813957 (+ 990us) service_pool.cc:225] Handling call
0112 02:52:39.647639 (+833682us) ts_tablet_manager.cc:1145] Deleting tablet 54215c55966a46e1ab96e352c4e3764e
0112 02:52:39.647649 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.707352 (+ 59703us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.750525 (+ 43173us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.815289 (+ 64764us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":833354}
I20250112 02:52:39.815949 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 54215c55966a46e1ab96e352c4e3764e (table default.Table_6e77252e817f43399c9b3f5b221795ad [id=1fa1474ab8804667b22e5f30c48c1630]) successfully deleted
I20250112 02:52:39.816138 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 171) took 1002 ms. Trace:
I20250112 02:52:39.816251 2562 rpcz_store.cc:276] 0112 02:52:38.813914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.814274 (+ 360us) service_pool.cc:225] Handling call
0112 02:52:39.815956 (+1001682us) ts_tablet_manager.cc:1145] Deleting tablet 83c04f1a2cb04950a9f6170e252c19b4
0112 02:52:39.815966 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.816121 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1001557}
I20250112 02:52:39.816694 2562 tablet_replica.cc:331] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:39.816924 2562 raft_consensus.cc:2238] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:39.817096 2562 raft_consensus.cc:2267] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.817476 1181 tablet_bootstrap.cc:492] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:39.817692 2562 ts_tablet_manager.cc:1905] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.817967 1177 raft_consensus.cc:383] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.818037 1177 raft_consensus.cc:738] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.818224 1177 consensus_queue.cc:260] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.818746 1177 ts_tablet_manager.cc:1428] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250112 02:52:39.818473 446 catalog_manager.cc:5526] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } }
I20250112 02:52:39.818953 1177 tablet_bootstrap.cc:492] T 9af55dab202444358ea1a0674d37946d P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:39.819355 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
W20250112 02:52:39.819335 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 83c04f1a2cb04950a9f6170e252c19b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83c04f1a2cb04950a9f6170e252c19b4 already in progress: creating tablet
I20250112 02:52:39.819798 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:39.820132 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:39.820379 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:39.821252 446 catalog_manager.cc:5526] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } }
I20250112 02:52:39.822091 446 catalog_manager.cc:5526] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } }
I20250112 02:52:39.823045 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:39.823901 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:39.824371 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
I20250112 02:52:39.824658 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:39.824992 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
W20250112 02:52:39.825623 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24ebf85f76f44633a20d97964638c0ac consensus state has no leader
I20250112 02:52:39.825780 448 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 146) took 3874 ms. Trace:
I20250112 02:52:39.825865 448 rpcz_store.cc:276] 0112 02:52:35.951000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.219999 (+3268999us) service_pool.cc:225] Handling call
0112 02:52:39.255028 (+ 35029us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:39.255261 (+ 233us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:39.255437 (+ 176us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:39.255521 (+ 84us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:39.326997 (+ 71476us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:39.825752 (+498755us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:39.256033 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:39.256140 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:39.256145 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:39.256148 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:39.256233 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:39.256242 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:39.256246 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:39.256297 (+ 51us) tablet.cc:698] Row locks acquired
0112 02:52:39.256300 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:39.256330 (+ 30us) write_op.cc:270] Start()
0112 02:52:39.256358 (+ 28us) write_op.cc:276] Timestamp: P: 1736650359256289 usec, L: 0
0112 02:52:39.256362 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:39.256545 (+ 183us) log.cc:844] Serialized 405 byte log entry
0112 02:52:39.326484 (+ 69939us) op_driver.cc:464] REPLICATION: finished
0112 02:52:39.326631 (+ 147us) write_op.cc:301] APPLY: starting
0112 02:52:39.326644 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:39.326681 (+ 37us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:39.326711 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:39.326759 (+ 48us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:39.326797 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:39.326803 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:39.326862 (+ 59us) log.cc:844] Serialized 23 byte log entry
0112 02:52:39.326912 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.326925 (+ 13us) write_op.cc:454] Released schema lock
0112 02:52:39.326931 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:39.326995 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.326998 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":106,"apply.run_cpu_time_us":356,"apply.run_wall_time_us":355,"num_ops":1,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":583,"prepare.run_wall_time_us":583,"replication_time_us":70095}]]}
I20250112 02:52:39.832129 1177 tablet_bootstrap.cc:654] T 9af55dab202444358ea1a0674d37946d P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.817654 1181 ts_tablet_manager.cc:1397] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.097s user 0.007s sys 0.000s
I20250112 02:52:39.833284 1181 raft_consensus.cc:357] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.833462 1181 raft_consensus.cc:383] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.833515 1181 raft_consensus.cc:738] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.833698 1181 consensus_queue.cc:260] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.834147 1181 ts_tablet_manager.cc:1428] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:39.834391 1181 tablet_bootstrap.cc:492] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:39.835690 1181 tablet_bootstrap.cc:654] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.839847 1173 tablet_bootstrap.cc:654] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.841740 2003 ts_tablet_manager.cc:1918] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:39.843536 2003 log.cc:1198] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/e5c10fad790046a88fbc8ebe0303c481
I20250112 02:52:39.843951 2003 ts_tablet_manager.cc:1939] T e5c10fad790046a88fbc8ebe0303c481 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:39.842671 1177 tablet_bootstrap.cc:492] T 9af55dab202444358ea1a0674d37946d P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:39.844988 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet e5c10fad790046a88fbc8ebe0303c481 (table default.Table_16b036d8833a47a99577ab67579ccb5a [id=eb980088c4584ccf82bb585c2210894a]) successfully deleted
I20250112 02:52:39.845038 1177 ts_tablet_manager.cc:1397] T 9af55dab202444358ea1a0674d37946d P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20250112 02:52:39.845743 1177 raft_consensus.cc:357] T 9af55dab202444358ea1a0674d37946d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.846007 1177 raft_consensus.cc:383] T 9af55dab202444358ea1a0674d37946d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.846094 1177 raft_consensus.cc:738] T 9af55dab202444358ea1a0674d37946d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.846338 1177 consensus_queue.cc:260] T 9af55dab202444358ea1a0674d37946d P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.846774 2598 raft_consensus.cc:491] T 709517ae520640208abe85fa14a5301e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.846882 2598 raft_consensus.cc:513] T 709517ae520640208abe85fa14a5301e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.847369 2598 leader_election.cc:290] T 709517ae520640208abe85fa14a5301e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:39.848263 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709517ae520640208abe85fa14a5301e" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:39.848831 855 leader_election.cc:343] T 709517ae520640208abe85fa14a5301e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.848886 2562 ts_tablet_manager.cc:1918] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:39.848994 2562 log.cc:1198] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/82b351c563d749df806ab5d0331a4ef2
I20250112 02:52:39.849344 1177 ts_tablet_manager.cc:1428] T 9af55dab202444358ea1a0674d37946d P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250112 02:52:39.849625 2562 ts_tablet_manager.cc:1939] T 82b351c563d749df806ab5d0331a4ef2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:39.849628 1177 tablet_bootstrap.cc:492] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:39.851148 1177 tablet_bootstrap.cc:654] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:39.857842 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet ea33e5c1a1fb46599c5836a36b130051 because the tablet was not found. No further retry: Not found: Tablet not found: ea33e5c1a1fb46599c5836a36b130051
W20250112 02:52:39.858462 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 because the tablet was not found. No further retry: Not found: Tablet not found: 6913b0ac9ceb4dbc9cdf591f159134f7
W20250112 02:52:39.858707 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 5dd48d0e68014e1aa0c44620527baec9 because the tablet was not found. No further retry: Not found: Tablet not found: 5dd48d0e68014e1aa0c44620527baec9
I20250112 02:52:39.859652 2003 tablet_replica.cc:331] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:39.859895 2003 raft_consensus.cc:2238] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:39.860504 2003 pending_rounds.cc:62] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
W20250112 02:52:39.860605 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 9d5b3df88a0c45dba879b3d506710614 because the tablet was not found. No further retry: Not found: Tablet not found: 9d5b3df88a0c45dba879b3d506710614
I20250112 02:52:39.860750 2003 pending_rounds.cc:69] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319872071065600 op_type: NO_OP noop_request { }
W20250112 02:52:39.860817 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 4b599e983ec04f53a440d1ab828d12ab because the tablet was not found. No further retry: Not found: Tablet not found: 4b599e983ec04f53a440d1ab828d12ab
I20250112 02:52:39.848258 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709517ae520640208abe85fa14a5301e" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:39.860265 1181 tablet_bootstrap.cc:492] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
W20250112 02:52:39.863338 854 leader_election.cc:343] T 709517ae520640208abe85fa14a5301e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.863451 854 leader_election.cc:304] T 709517ae520640208abe85fa14a5301e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:39.863629 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 159) took 1051 ms. Trace:
I20250112 02:52:39.863691 2562 rpcz_store.cc:276] 0112 02:52:38.812631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.814546 (+ 1915us) service_pool.cc:225] Handling call
0112 02:52:39.816637 (+1002091us) ts_tablet_manager.cc:1145] Deleting tablet 82b351c563d749df806ab5d0331a4ef2
0112 02:52:39.816662 (+ 25us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.845862 (+ 29200us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.848802 (+ 2940us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.863617 (+ 14815us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1001971}
I20250112 02:52:39.863807 2598 raft_consensus.cc:2743] T 709517ae520640208abe85fa14a5301e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:39.864084 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 82b351c563d749df806ab5d0331a4ef2 (table default.Table_4603068c01544e2e965dca5177e36b40 [id=8502f21e9b61436093db1241cfbb5ec9]) successfully deleted
I20250112 02:52:39.865550 2003 raft_consensus.cc:2883] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:39.865660 2003 raft_consensus.cc:2267] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.865846 2024 ts_tablet_manager.cc:1918] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:39.866564 2024 log.cc:1198] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/daef3a58605e40978f7389698b22265d
I20250112 02:52:39.871898 2024 ts_tablet_manager.cc:1939] T daef3a58605e40978f7389698b22265d P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:39.875916 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 82) took 5342 ms. Trace:
I20250112 02:52:39.863338 1181 ts_tablet_manager.cc:1397] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20250112 02:52:39.876828 1181 raft_consensus.cc:357] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:39.877010 1181 raft_consensus.cc:383] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.877061 1181 raft_consensus.cc:738] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.876732 2003 ts_tablet_manager.cc:1905] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.877229 1181 consensus_queue.cc:260] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:39.877663 1181 ts_tablet_manager.cc:1428] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 02:52:39.877913 1181 tablet_bootstrap.cc:492] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:39.878968 1177 tablet_bootstrap.cc:492] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:39.879087 1177 ts_tablet_manager.cc:1397] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250112 02:52:39.879246 1181 tablet_bootstrap.cc:654] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.880206 1177 raft_consensus.cc:357] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.880383 1177 raft_consensus.cc:383] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.880434 1177 raft_consensus.cc:738] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.880613 1177 consensus_queue.cc:260] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.881122 1177 ts_tablet_manager.cc:1428] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:39.886204 1177 tablet_bootstrap.cc:492] T a437952889494b0e820e41fdfffe708d P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:39.864239 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 160) took 1051 ms. Trace:
I20250112 02:52:39.884785 2609 raft_consensus.cc:491] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.891916 2609 raft_consensus.cc:513] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.892526 2609 leader_election.cc:290] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.887430 2326 consensus_queue.cc:1035] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:39.893270 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4348fe4c32704bf7b3a8d21ff3b8ac1b" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:39.894021 779 raft_consensus.cc:2463] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:39.894416 991 leader_election.cc:304] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:39.894763 2326 raft_consensus.cc:2798] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:39.894869 2326 raft_consensus.cc:491] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:39.894989 2326 raft_consensus.cc:3054] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.896046 2179 raft_consensus.cc:491] T bb4ef0220dde43569452379639759db5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.896158 2179 raft_consensus.cc:513] T bb4ef0220dde43569452379639759db5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.896685 2179 leader_election.cc:290] T bb4ef0220dde43569452379639759db5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.897420 2326 raft_consensus.cc:513] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.898267 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4348fe4c32704bf7b3a8d21ff3b8ac1b" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:39.898408 779 raft_consensus.cc:3054] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.898972 1177 tablet_bootstrap.cc:654] T a437952889494b0e820e41fdfffe708d P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.900821 779 raft_consensus.cc:2463] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:39.901350 991 leader_election.cc:304] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:39.902173 2609 raft_consensus.cc:2798] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:39.902304 2609 raft_consensus.cc:695] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:39.902554 2609 consensus_queue.cc:237] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:39.904217 989 leader_election.cc:343] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 4348fe4c32704bf7b3a8d21ff3b8ac1b
W20250112 02:52:39.904695 989 leader_election.cc:343] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 4348fe4c32704bf7b3a8d21ff3b8ac1b
I20250112 02:52:39.905120 2326 leader_election.cc:290] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:39.905736 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4ef0220dde43569452379639759db5" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:39.907140 779 raft_consensus.cc:3054] T bb4ef0220dde43569452379639759db5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:39.910229 2208 consensus_queue.cc:1035] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:39.888585 1173 tablet_bootstrap.cc:492] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:39.916188 1173 ts_tablet_manager.cc:1397] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.115s user 0.003s sys 0.001s
I20250112 02:52:39.916875 1173 raft_consensus.cc:357] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.917057 1173 raft_consensus.cc:383] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.917138 1173 raft_consensus.cc:738] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.917346 1173 consensus_queue.cc:260] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.920166 1173 ts_tablet_manager.cc:1428] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 02:52:39.920457 1173 tablet_bootstrap.cc:492] T 06edfca60eed4dd3a2dee7b662b0970c P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:39.927592 2024 rpcz_store.cc:276] 0112 02:52:34.533733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.533788 (+ 55us) service_pool.cc:225] Handling call
0112 02:52:39.777023 (+5243235us) ts_tablet_manager.cc:1145] Deleting tablet daef3a58605e40978f7389698b22265d
0112 02:52:39.777032 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.804950 (+ 27918us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.860541 (+ 55591us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:39.875850 (+ 15309us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5234151}
I20250112 02:52:39.928443 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet daef3a58605e40978f7389698b22265d (table default.Table_8e00e020edc14465ad0e935be514d03d [id=cf8528b7c33a47ce984da87382bef6a5]) successfully deleted
I20250112 02:52:39.928906 2024 tablet_replica.cc:331] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:39.929203 2024 raft_consensus.cc:2238] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:39.929606 2024 raft_consensus.cc:2267] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.930261 2024 ts_tablet_manager.cc:1905] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.932757 2003 ts_tablet_manager.cc:1918] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:39.933162 2003 log.cc:1198] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/b6d84027ad6d40a5be5afcdd854431a5
I20250112 02:52:39.934296 2003 ts_tablet_manager.cc:1939] T b6d84027ad6d40a5be5afcdd854431a5 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:39.935487 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet b6d84027ad6d40a5be5afcdd854431a5 (table default.Table_31424b670f274fd3abd17eb8451ba95c [id=ebc9aae273fb4bcd8bd904f2179f1dd4]) successfully deleted
W20250112 02:52:39.936520 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet a85300a9aa1c4c50a3957a004bea7933 because the tablet was not found. No further retry: Not found: Tablet not found: a85300a9aa1c4c50a3957a004bea7933
W20250112 02:52:39.937651 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 352c87f1dc724e7b94bc2e64abfce8d7 because the tablet was not found. No further retry: Not found: Tablet not found: 352c87f1dc724e7b94bc2e64abfce8d7
I20250112 02:52:39.937995 2003 tablet_replica.cc:331] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:39.938730 2003 raft_consensus.cc:2238] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:39.939273 2003 pending_rounds.cc:62] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:39.939496 2003 pending_rounds.cc:69] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319871881375744 op_type: NO_OP noop_request { }
I20250112 02:52:39.939774 2003 raft_consensus.cc:2883] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:39.940053 2003 raft_consensus.cc:2267] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.897727 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4ef0220dde43569452379639759db5" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:39.941578 713 leader_election.cc:304] T bb4ef0220dde43569452379639759db5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:39.941987 2047 raft_consensus.cc:2743] T bb4ef0220dde43569452379639759db5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:39.942153 1177 tablet_bootstrap.cc:492] T a437952889494b0e820e41fdfffe708d P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:39.942281 1177 ts_tablet_manager.cc:1397] T a437952889494b0e820e41fdfffe708d P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.056s user 0.007s sys 0.000s
I20250112 02:52:39.942171 2562 rpcz_store.cc:276] 0112 02:52:38.812729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.814693 (+ 1964us) service_pool.cc:225] Handling call
0112 02:52:39.864148 (+1049455us) ts_tablet_manager.cc:1145] Deleting tablet 1a1e3af2883341e9b957f6ed748e604d
0112 02:52:39.864158 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.864223 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1049323}
W20250112 02:52:39.942561 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 1a1e3af2883341e9b957f6ed748e604d because the tablet was not found. No further retry: Not found: Tablet not found: 1a1e3af2883341e9b957f6ed748e604d
I20250112 02:52:39.942651 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 165) took 1129 ms. Trace:
I20250112 02:52:39.942728 2562 rpcz_store.cc:276] 0112 02:52:38.813302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.814853 (+ 1551us) service_pool.cc:225] Handling call
0112 02:52:39.942560 (+1127707us) ts_tablet_manager.cc:1145] Deleting tablet 1a8a178bafcb457f9b0e9587df560493
0112 02:52:39.942569 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.942634 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1127582}
I20250112 02:52:39.942574 2003 ts_tablet_manager.cc:1905] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.918292 1181 tablet_bootstrap.cc:492] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:39.943050 1177 raft_consensus.cc:357] T a437952889494b0e820e41fdfffe708d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.943172 1181 ts_tablet_manager.cc:1397] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.065s user 0.006s sys 0.002s
W20250112 02:52:39.943158 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 1a8a178bafcb457f9b0e9587df560493 because the tablet was not found. No further retry: Not found: Tablet not found: 1a8a178bafcb457f9b0e9587df560493
I20250112 02:52:39.943225 1177 raft_consensus.cc:383] T a437952889494b0e820e41fdfffe708d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.943275 1177 raft_consensus.cc:738] T a437952889494b0e820e41fdfffe708d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.943503 1177 consensus_queue.cc:260] T a437952889494b0e820e41fdfffe708d P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.944244 2598 raft_consensus.cc:491] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.944458 2598 raft_consensus.cc:513] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:39.945204 2598 leader_election.cc:290] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:39.945952 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7340a7a916c4b2fab32e0ba2d0ce322" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:39.946511 855 leader_election.cc:343] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.946579 1177 ts_tablet_manager.cc:1428] T a437952889494b0e820e41fdfffe708d P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:39.946847 1177 tablet_bootstrap.cc:492] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:39.947197 854 leader_election.cc:343] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: b7340a7a916c4b2fab32e0ba2d0ce322
I20250112 02:52:39.947331 854 leader_election.cc:304] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:39.947741 2598 raft_consensus.cc:491] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:39.947851 2598 raft_consensus.cc:513] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.948377 2598 leader_election.cc:290] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:39.948493 2598 raft_consensus.cc:2743] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:39.949599 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f4bd2df0ce45dabc55373f1154d3ac" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:39.950206 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f4bd2df0ce45dabc55373f1154d3ac" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:39.950596 855 leader_election.cc:343] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:39.950947 854 leader_election.cc:343] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:39.951031 854 leader_election.cc:304] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:39.944115 1181 raft_consensus.cc:357] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.953675 1181 raft_consensus.cc:383] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.953769 1181 raft_consensus.cc:738] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.954075 1181 consensus_queue.cc:260] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.954097 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:39.954389 2598 raft_consensus.cc:2743] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:39.954882 1181 ts_tablet_manager.cc:1428] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250112 02:52:39.955049 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:39.955272 1181 tablet_bootstrap.cc:492] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:39.955909 1177 tablet_bootstrap.cc:654] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.956694 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:39.956961 1181 tablet_bootstrap.cc:654] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.957319 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:39.958097 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
I20250112 02:52:39.959060 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:39.959750 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 172) took 1145 ms. Trace:
I20250112 02:52:39.959872 2562 rpcz_store.cc:276] 0112 02:52:38.814505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.815014 (+ 509us) service_pool.cc:225] Handling call
0112 02:52:39.955025 (+1140011us) ts_tablet_manager.cc:1145] Deleting tablet 13578542e9e741e8817e0ee63b10f1a1
0112 02:52:39.959649 (+ 4624us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.959735 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1139900}
I20250112 02:52:39.960350 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 173) took 1145 ms. Trace:
W20250112 02:52:39.960526 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 13578542e9e741e8817e0ee63b10f1a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13578542e9e741e8817e0ee63b10f1a1 already in progress: creating tablet
I20250112 02:52:39.960464 2562 rpcz_store.cc:276] 0112 02:52:38.815229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.815745 (+ 516us) service_pool.cc:225] Handling call
0112 02:52:39.960281 (+1144536us) ts_tablet_manager.cc:1145] Deleting tablet 04f135b226774285a077e5806131d91b
0112 02:52:39.960289 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.960336 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1144398}
W20250112 02:52:39.960798 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 04f135b226774285a077e5806131d91b because the tablet was not found. No further retry: Not found: Tablet not found: 04f135b226774285a077e5806131d91b
I20250112 02:52:39.960815 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 174) took 1144 ms. Trace:
I20250112 02:52:39.960868 2562 rpcz_store.cc:276] 0112 02:52:38.815992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.816215 (+ 223us) service_pool.cc:225] Handling call
0112 02:52:39.960748 (+1144533us) ts_tablet_manager.cc:1145] Deleting tablet 161fe26b9c0e4543b7aeb0021506d72a
0112 02:52:39.960757 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.960803 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1144415}
I20250112 02:52:39.961212 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 175) took 1145 ms. Trace:
I20250112 02:52:39.961269 2562 rpcz_store.cc:276] 0112 02:52:38.816078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.816362 (+ 284us) service_pool.cc:225] Handling call
0112 02:52:39.961154 (+1144792us) ts_tablet_manager.cc:1145] Deleting tablet 416fb6820cd54e118950c68d381f51b2
0112 02:52:39.961161 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.961200 (+ 39us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1144687}
I20250112 02:52:39.961580 2562 tablet_replica.cc:331] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:39.961766 2562 raft_consensus.cc:2238] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:39.961884 2562 raft_consensus.cc:2267] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.961714 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:39.962368 2562 ts_tablet_manager.cc:1905] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.962646 2003 ts_tablet_manager.cc:1918] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:39.962955 2003 log.cc:1198] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/7a34d03a80ae402db6607261a5c40b60
W20250112 02:52:39.963212 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 161fe26b9c0e4543b7aeb0021506d72a because the tablet was not found. No further retry: Not found: Tablet not found: 161fe26b9c0e4543b7aeb0021506d72a
W20250112 02:52:39.963457 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 416fb6820cd54e118950c68d381f51b2 because the tablet was not found. No further retry: Not found: Tablet not found: 416fb6820cd54e118950c68d381f51b2
I20250112 02:52:39.964283 2003 ts_tablet_manager.cc:1939] T 7a34d03a80ae402db6607261a5c40b60 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:39.964221 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:39.965863 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 7a34d03a80ae402db6607261a5c40b60 (table default.Table_af61980bca5e4995a77da652dcfb3f0d [id=9d7284a74b4148c0a8d20eb220ef9e17]) successfully deleted
I20250112 02:52:39.966383 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
W20250112 02:52:39.967141 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 9163878cfeb64693a9c5493e812546af because the tablet was not found. No further retry: Not found: Tablet not found: 9163878cfeb64693a9c5493e812546af
I20250112 02:52:39.967783 2003 tablet_replica.cc:331] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:39.967679 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
I20250112 02:52:39.968444 2003 raft_consensus.cc:2238] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:39.969455 2003 raft_consensus.cc:2267] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:39.970500 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 1 got 0
I20250112 02:52:39.971611 2003 ts_tablet_manager.cc:1905] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:39.972843 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:39.973204 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 6117dc8b32a64535a2f4167a03bed0d2 (table default.Table_365739770a11434dbf260ea912e7ea81 [id=868adb627654450c936ef198e49d853b]). Expected version 1 got 0
I20250112 02:52:39.980330 1177 tablet_bootstrap.cc:492] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:39.980533 1177 ts_tablet_manager.cc:1397] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20250112 02:52:39.981628 1177 raft_consensus.cc:357] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.981798 1173 tablet_bootstrap.cc:654] T 06edfca60eed4dd3a2dee7b662b0970c P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:39.981933 1177 raft_consensus.cc:383] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.982033 1177 raft_consensus.cc:738] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.982295 1177 consensus_queue.cc:260] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.987119 1177 ts_tablet_manager.cc:1428] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250112 02:52:39.988071 1177 tablet_bootstrap.cc:492] T edfd1ebe729d458ea31eb7feff7ee072 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:39.987775 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 09d06e2e8c13464db8cf801a292bdc99 (table default.Table_2c52fead633649dd869763e5aa532be7 [id=e3bb489c53454222a79702b798c0c2f1]). Expected version 3 got 2
I20250112 02:52:39.990480 1173 tablet_bootstrap.cc:492] T 06edfca60eed4dd3a2dee7b662b0970c P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:39.990636 1173 ts_tablet_manager.cc:1397] T 06edfca60eed4dd3a2dee7b662b0970c P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.002s
I20250112 02:52:39.991338 1173 raft_consensus.cc:357] T 06edfca60eed4dd3a2dee7b662b0970c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.991575 1173 raft_consensus.cc:383] T 06edfca60eed4dd3a2dee7b662b0970c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:39.991668 1173 raft_consensus.cc:738] T 06edfca60eed4dd3a2dee7b662b0970c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:39.991945 1173 consensus_queue.cc:260] T 06edfca60eed4dd3a2dee7b662b0970c P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:39.991685 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:39.995219 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:39.996466 1173 ts_tablet_manager.cc:1428] T 06edfca60eed4dd3a2dee7b662b0970c P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250112 02:52:39.997804 1173 tablet_bootstrap.cc:492] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:39.998971 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet fccff64af8ca4400afc116db98918119 (table default.Table_15bcb2c0f2a74458a1bed231fd3437fc [id=ca9c8281bf254f248076cb600e7c4f87]). Expected version 2 got 0
I20250112 02:52:40.000341 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:40.000902 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:40.002614 1177 tablet_bootstrap.cc:654] T edfd1ebe729d458ea31eb7feff7ee072 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.002465 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:40.004031 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:40.004864 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:40.005577 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 5cc414d6420c41839621ab858067548b (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]). Expected version 1 got 0
I20250112 02:52:40.006223 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:40.007339 2003 ts_tablet_manager.cc:1918] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:40.007921 2003 log.cc:1198] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/1dda52f012794493a102ed1abc718f5b
I20250112 02:52:40.009444 2003 ts_tablet_manager.cc:1939] T 1dda52f012794493a102ed1abc718f5b P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:39.999439 1173 tablet_bootstrap.cc:654] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.010267 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:40.011116 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:40.011631 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:40.012281 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:40.012840 451 catalog_manager.cc:5526] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:40.020118 2024 ts_tablet_manager.cc:1918] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:40.020237 2024 log.cc:1198] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/9163878cfeb64693a9c5493e812546af
I20250112 02:52:40.020586 2024 ts_tablet_manager.cc:1939] T 9163878cfeb64693a9c5493e812546af P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:40.022346 2562 ts_tablet_manager.cc:1918] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:40.024423 2609 consensus_queue.cc:1035] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:40.022708 1181 tablet_bootstrap.cc:492] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:40.035653 1181 ts_tablet_manager.cc:1397] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.081s user 0.004s sys 0.000s
I20250112 02:52:40.036418 1181 raft_consensus.cc:357] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.036662 1181 raft_consensus.cc:383] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.036761 1181 raft_consensus.cc:738] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.037081 1181 consensus_queue.cc:260] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.044659 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 235) took 1068 ms. Trace:
I20250112 02:52:40.044745 2003 rpcz_store.cc:276] 0112 02:52:38.976511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.976561 (+ 50us) service_pool.cc:225] Handling call
0112 02:52:39.967620 (+991059us) ts_tablet_manager.cc:1145] Deleting tablet 1dda52f012794493a102ed1abc718f5b
0112 02:52:39.967649 (+ 29us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.996216 (+ 28567us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.007177 (+ 10961us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.044640 (+ 37463us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":988210}
I20250112 02:52:40.045094 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 234) took 1068 ms. Trace:
I20250112 02:52:40.045153 2003 rpcz_store.cc:276] 0112 02:52:38.976420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.976504 (+ 84us) service_pool.cc:225] Handling call
0112 02:52:40.045013 (+1068509us) ts_tablet_manager.cc:1145] Deleting tablet 04f135b226774285a077e5806131d91b
0112 02:52:40.045021 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.045080 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1058760}
I20250112 02:52:40.045423 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 233) took 1069 ms. Trace:
I20250112 02:52:40.045480 2003 rpcz_store.cc:276] 0112 02:52:38.976340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.976545 (+ 205us) service_pool.cc:225] Handling call
0112 02:52:40.045340 (+1068795us) ts_tablet_manager.cc:1145] Deleting tablet 1b184806b4884cd3b0a002905a1e57af
0112 02:52:40.045366 (+ 26us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.045411 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1057635}
I20250112 02:52:40.045722 2003 tablet_replica.cc:331] T 9af55dab202444358ea1a0674d37946d P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:40.045897 2003 raft_consensus.cc:2238] T 9af55dab202444358ea1a0674d37946d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:40.046037 2003 raft_consensus.cc:2267] T 9af55dab202444358ea1a0674d37946d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:40.046569 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 1dda52f012794493a102ed1abc718f5b (table default.Table_ab4d7c58924a4b8682b52637c23f125b [id=a622f7a973344a388b3275adbcce6234]) successfully deleted
W20250112 02:52:40.046839 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 04f135b226774285a077e5806131d91b because the tablet was not found. No further retry: Not found: Tablet not found: 04f135b226774285a077e5806131d91b
W20250112 02:52:40.047029 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 1b184806b4884cd3b0a002905a1e57af because the tablet was not found. No further retry: Not found: Tablet not found: 1b184806b4884cd3b0a002905a1e57af
I20250112 02:52:40.048633 1181 ts_tablet_manager.cc:1428] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20250112 02:52:40.049703 2003 ts_tablet_manager.cc:1905] T 9af55dab202444358ea1a0674d37946d P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.054040 1181 tablet_bootstrap.cc:492] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:40.055722 1181 tablet_bootstrap.cc:654] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.027266 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 83) took 5493 ms. Trace:
I20250112 02:52:40.056043 2024 rpcz_store.cc:276] 0112 02:52:34.533857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.533904 (+ 47us) service_pool.cc:225] Handling call
0112 02:52:39.928786 (+5394882us) ts_tablet_manager.cc:1145] Deleting tablet 9163878cfeb64693a9c5493e812546af
0112 02:52:39.928800 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.986961 (+ 58161us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.020010 (+ 33049us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.027228 (+ 7218us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5385847}
I20250112 02:52:40.056502 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 84) took 5522 ms. Trace:
I20250112 02:52:40.056556 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 9163878cfeb64693a9c5493e812546af (table default.Table_426c5d1d19a94f0d8110a9cbee9a9044 [id=69f55e532f714a2d84790df8ff19e6dc]) successfully deleted
I20250112 02:52:40.055869 2562 log.cc:1198] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/62f4bd2df0ce45dabc55373f1154d3ac
I20250112 02:52:40.060130 2562 ts_tablet_manager.cc:1939] T 62f4bd2df0ce45dabc55373f1154d3ac P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:40.064642 2024 rpcz_store.cc:276] 0112 02:52:34.534207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.534258 (+ 51us) service_pool.cc:225] Handling call
0112 02:52:40.056400 (+5522142us) ts_tablet_manager.cc:1145] Deleting tablet 5dd48d0e68014e1aa0c44620527baec9
0112 02:52:40.056409 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.056472 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5513425}
I20250112 02:52:40.065142 2024 tablet_replica.cc:331] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
W20250112 02:52:40.065308 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 5dd48d0e68014e1aa0c44620527baec9 because the tablet was not found. No further retry: Not found: Tablet not found: 5dd48d0e68014e1aa0c44620527baec9
I20250112 02:52:40.065425 2024 raft_consensus.cc:2238] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:40.065119 2003 ts_tablet_manager.cc:1918] T 9af55dab202444358ea1a0674d37946d P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:40.065769 2024 raft_consensus.cc:2267] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:40.066040 2003 log.cc:1198] T 9af55dab202444358ea1a0674d37946d P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/9af55dab202444358ea1a0674d37946d
I20250112 02:52:40.066427 1177 tablet_bootstrap.cc:492] T edfd1ebe729d458ea31eb7feff7ee072 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:40.066540 2024 ts_tablet_manager.cc:1905] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.066584 1177 ts_tablet_manager.cc:1397] T edfd1ebe729d458ea31eb7feff7ee072 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.079s user 0.010s sys 0.000s
I20250112 02:52:40.067373 1177 raft_consensus.cc:357] T edfd1ebe729d458ea31eb7feff7ee072 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.067924 1177 raft_consensus.cc:383] T edfd1ebe729d458ea31eb7feff7ee072 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.068068 1177 raft_consensus.cc:738] T edfd1ebe729d458ea31eb7feff7ee072 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.068367 1173 tablet_bootstrap.cc:492] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:40.068472 1177 consensus_queue.cc:260] T edfd1ebe729d458ea31eb7feff7ee072 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.068940 1173 ts_tablet_manager.cc:1397] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.071s user 0.008s sys 0.014s
I20250112 02:52:40.069204 1177 ts_tablet_manager.cc:1428] T edfd1ebe729d458ea31eb7feff7ee072 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 02:52:40.067390 827 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:40.067289 2003 ts_tablet_manager.cc:1939] T 9af55dab202444358ea1a0674d37946d P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:40.070564 2326 consensus_queue.cc:1035] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:40.071385 1177 tablet_bootstrap.cc:492] T 47a0d939fdb94590a86389f1f4693e9a P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:40.072479 2047 raft_consensus.cc:491] T edfd1ebe729d458ea31eb7feff7ee072 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.072607 2047 raft_consensus.cc:513] T edfd1ebe729d458ea31eb7feff7ee072 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.073192 2047 leader_election.cc:290] T edfd1ebe729d458ea31eb7feff7ee072 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.073580 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfd1ebe729d458ea31eb7feff7ee072" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:40.074070 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfd1ebe729d458ea31eb7feff7ee072" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:40.074357 917 raft_consensus.cc:2388] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 289afc116aa74408aa5bff74e11c50d8 in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
I20250112 02:52:40.075153 714 leader_election.cc:304] T edfd1ebe729d458ea31eb7feff7ee072 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:40.075435 2047 raft_consensus.cc:3054] T edfd1ebe729d458ea31eb7feff7ee072 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:40.077381 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 158) took 1264 ms. Trace:
I20250112 02:52:40.077641 2047 raft_consensus.cc:2743] T edfd1ebe729d458ea31eb7feff7ee072 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:40.077527 2562 rpcz_store.cc:276] 0112 02:52:38.812530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.817787 (+ 5257us) service_pool.cc:225] Handling call
0112 02:52:39.961544 (+1143757us) ts_tablet_manager.cc:1145] Deleting tablet 62f4bd2df0ce45dabc55373f1154d3ac
0112 02:52:39.961552 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:39.965528 (+ 3976us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.022259 (+ 56731us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.077360 (+ 55101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1143623}
I20250112 02:52:40.077924 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 62f4bd2df0ce45dabc55373f1154d3ac (table default.Table_7921b99e24f84cceb58128f18925ad49 [id=0525f87eccbb4a1d8726d18a00f9a4e4]) successfully deleted
I20250112 02:52:40.078008 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 177) took 1250 ms. Trace:
I20250112 02:52:40.078066 2562 rpcz_store.cc:276] 0112 02:52:38.827920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.828105 (+ 185us) service_pool.cc:225] Handling call
0112 02:52:40.077921 (+1249816us) ts_tablet_manager.cc:1145] Deleting tablet 49bb6d20cfa0453e90e048825114525e
0112 02:52:40.077929 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.077987 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1249657}
I20250112 02:52:40.078444 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 178) took 1250 ms. Trace:
W20250112 02:52:40.078428 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 49bb6d20cfa0453e90e048825114525e because the tablet was not found. No further retry: Not found: Tablet not found: 49bb6d20cfa0453e90e048825114525e
I20250112 02:52:40.078497 2562 rpcz_store.cc:276] 0112 02:52:38.828405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.828592 (+ 187us) service_pool.cc:225] Handling call
0112 02:52:40.078380 (+1249788us) ts_tablet_manager.cc:1145] Deleting tablet 41e166b3f8414ababbd4a3f07fb90c4c
0112 02:52:40.078387 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.078432 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1249623}
W20250112 02:52:40.078815 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 41e166b3f8414ababbd4a3f07fb90c4c because the tablet was not found. No further retry: Not found: Tablet not found: 41e166b3f8414ababbd4a3f07fb90c4c
I20250112 02:52:40.078861 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 179) took 1250 ms. Trace:
I20250112 02:52:40.078913 2562 rpcz_store.cc:276] 0112 02:52:38.828879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.829444 (+ 565us) service_pool.cc:225] Handling call
0112 02:52:40.078796 (+1249352us) ts_tablet_manager.cc:1145] Deleting tablet 7f4e4f1e451b48dc9cdd780300f6888c
0112 02:52:40.078806 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.078849 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1249217}
W20250112 02:52:40.079252 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 7f4e4f1e451b48dc9cdd780300f6888c because the tablet was not found. No further retry: Not found: Tablet not found: 7f4e4f1e451b48dc9cdd780300f6888c
I20250112 02:52:40.079293 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 180) took 1249 ms. Trace:
I20250112 02:52:40.079345 2562 rpcz_store.cc:276] 0112 02:52:38.829710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.829874 (+ 164us) service_pool.cc:225] Handling call
0112 02:52:40.079227 (+1249353us) ts_tablet_manager.cc:1145] Deleting tablet fe614eaeb793473d9b2da206b7345562
0112 02:52:40.079237 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.079281 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1249232}
W20250112 02:52:40.080190 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet fe614eaeb793473d9b2da206b7345562 because the tablet was not found. No further retry: Not found: Tablet not found: fe614eaeb793473d9b2da206b7345562
I20250112 02:52:40.085759 1177 tablet_bootstrap.cc:654] T 47a0d939fdb94590a86389f1f4693e9a P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.086342 1173 raft_consensus.cc:357] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.087338 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 181) took 1257 ms. Trace:
I20250112 02:52:40.087479 2562 rpcz_store.cc:276] 0112 02:52:38.830133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.830440 (+ 307us) service_pool.cc:225] Handling call
0112 02:52:40.079692 (+1249252us) ts_tablet_manager.cc:1145] Deleting tablet 70715db0394d46d9b6b289160b00be7d
0112 02:52:40.079699 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.087318 (+ 7619us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1249101}
W20250112 02:52:40.087977 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 70715db0394d46d9b6b289160b00be7d because the tablet was not found. No further retry: Not found: Tablet not found: 70715db0394d46d9b6b289160b00be7d
I20250112 02:52:40.088025 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 182) took 1257 ms. Trace:
I20250112 02:52:40.088881 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:40.089066 2562 rpcz_store.cc:276] 0112 02:52:38.830728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.831063 (+ 335us) service_pool.cc:225] Handling call
0112 02:52:40.087937 (+1256874us) ts_tablet_manager.cc:1145] Deleting tablet 87759e8b99b04e628badc8f60cb38bb3
0112 02:52:40.087947 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.088011 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1256731}
I20250112 02:52:40.089354 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
W20250112 02:52:40.089449 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 87759e8b99b04e628badc8f60cb38bb3 because the tablet was not found. No further retry: Not found: Tablet not found: 87759e8b99b04e628badc8f60cb38bb3
I20250112 02:52:40.090163 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:40.091269 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:40.091730 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 237) took 1088 ms. Trace:
I20250112 02:52:40.091836 2003 rpcz_store.cc:276] 0112 02:52:39.003605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.003660 (+ 55us) service_pool.cc:225] Handling call
0112 02:52:40.045677 (+1042017us) ts_tablet_manager.cc:1145] Deleting tablet 9af55dab202444358ea1a0674d37946d
0112 02:52:40.045687 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.054092 (+ 8405us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.064781 (+ 10689us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.091711 (+ 26930us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":309632,"tablet-delete.queue_time_us":1041870}
I20250112 02:52:40.092247 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 238) took 1074 ms. Trace:
I20250112 02:52:40.092255 1173 raft_consensus.cc:383] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.092357 1173 raft_consensus.cc:738] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.092320 2003 rpcz_store.cc:276] 0112 02:52:39.017430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.017480 (+ 50us) service_pool.cc:225] Handling call
0112 02:52:40.092160 (+1074680us) ts_tablet_manager.cc:1145] Deleting tablet 73136be25c79491784d3ea71aa9b27f2
0112 02:52:40.092169 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.092232 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1071728}
I20250112 02:52:40.092412 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 9af55dab202444358ea1a0674d37946d (table default.Table_26dfe346379948b39f90b31653c2a463 [id=6c6f30e60e79451397841d090d11d8b7]) successfully deleted
I20250112 02:52:40.092604 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 240) took 1068 ms. Trace:
I20250112 02:52:40.092550 1173 consensus_queue.cc:260] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.092679 2003 rpcz_store.cc:276] 0112 02:52:39.023701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.024178 (+ 477us) service_pool.cc:225] Handling call
0112 02:52:40.092530 (+1068352us) ts_tablet_manager.cc:1145] Deleting tablet d4abf59d96414e5faa7cf75ba133832c
0112 02:52:40.092539 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.092589 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1068206}
I20250112 02:52:40.092968 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 241) took 1069 ms. Trace:
I20250112 02:52:40.093025 2003 rpcz_store.cc:276] 0112 02:52:39.023804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.024369 (+ 565us) service_pool.cc:225] Handling call
0112 02:52:40.092885 (+1068516us) ts_tablet_manager.cc:1145] Deleting tablet c8a7281c0c09442ebd0cc13b8496c86e
0112 02:52:40.092894 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.092942 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1068397}
I20250112 02:52:40.093277 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 239) took 1069 ms. Trace:
W20250112 02:52:40.093317 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 73136be25c79491784d3ea71aa9b27f2 because the tablet was not found. No further retry: Not found: Tablet not found: 73136be25c79491784d3ea71aa9b27f2
I20250112 02:52:40.093346 2003 rpcz_store.cc:276] 0112 02:52:39.023612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.024617 (+ 1005us) service_pool.cc:225] Handling call
0112 02:52:40.093213 (+1068596us) ts_tablet_manager.cc:1145] Deleting tablet 6cb76746b3a846c3aba531c04bf63966
0112 02:52:40.093222 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.093265 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1068466}
I20250112 02:52:40.093451 1173 ts_tablet_manager.cc:1428] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
W20250112 02:52:40.093535 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d4abf59d96414e5faa7cf75ba133832c because the tablet was not found. No further retry: Not found: Tablet not found: d4abf59d96414e5faa7cf75ba133832c
I20250112 02:52:40.093583 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 243) took 1062 ms. Trace:
I20250112 02:52:40.093633 2003 rpcz_store.cc:276] 0112 02:52:39.031267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.031386 (+ 119us) service_pool.cc:225] Handling call
0112 02:52:40.093520 (+1062134us) ts_tablet_manager.cc:1145] Deleting tablet 7a18132549374218893f35a4e4226b72
0112 02:52:40.093530 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.093571 (+ 41us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1061925}
I20250112 02:52:40.093886 1173 tablet_bootstrap.cc:492] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:40.093886 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 244) took 1062 ms. Trace:
I20250112 02:52:40.093945 2003 rpcz_store.cc:276] 0112 02:52:39.031347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.031643 (+ 296us) service_pool.cc:225] Handling call
0112 02:52:40.093823 (+1062180us) ts_tablet_manager.cc:1145] Deleting tablet b8daf4ef3593410eae87672402292be6
0112 02:52:40.093831 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.093874 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1062049}
W20250112 02:52:40.094158 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet c8a7281c0c09442ebd0cc13b8496c86e because the tablet was not found. No further retry: Not found: Tablet not found: c8a7281c0c09442ebd0cc13b8496c86e
I20250112 02:52:40.094238 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 242) took 1063 ms. Trace:
I20250112 02:52:40.094290 2003 rpcz_store.cc:276] 0112 02:52:39.031146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.031875 (+ 729us) service_pool.cc:225] Handling call
0112 02:52:40.094172 (+1062297us) ts_tablet_manager.cc:1145] Deleting tablet d45fce86e02f46d4a57af718fa2a62c8
0112 02:52:40.094180 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.094225 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1062165}
W20250112 02:52:40.094359 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 6cb76746b3a846c3aba531c04bf63966 because the tablet was not found. No further retry: Not found: Tablet not found: 6cb76746b3a846c3aba531c04bf63966
W20250112 02:52:40.094533 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 7a18132549374218893f35a4e4226b72 because the tablet was not found. No further retry: Not found: Tablet not found: 7a18132549374218893f35a4e4226b72
I20250112 02:52:40.094552 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 246) took 1061 ms. Trace:
I20250112 02:52:40.094604 2003 rpcz_store.cc:276] 0112 02:52:39.032906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.032947 (+ 41us) service_pool.cc:225] Handling call
0112 02:52:40.094463 (+1061516us) ts_tablet_manager.cc:1145] Deleting tablet 8c821ea3ca4946c290c17135366e0cac
0112 02:52:40.094472 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.094537 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1061375}
I20250112 02:52:40.094844 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 245) took 1062 ms. Trace:
I20250112 02:52:40.094909 2003 rpcz_store.cc:276] 0112 02:52:39.032795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.033158 (+ 363us) service_pool.cc:225] Handling call
0112 02:52:40.094782 (+1061624us) ts_tablet_manager.cc:1145] Deleting tablet f234256b343f447dadfc3d1936a824ee
0112 02:52:40.094789 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.094830 (+ 41us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1061501}
I20250112 02:52:40.095166 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 249) took 1061 ms. Trace:
I20250112 02:52:40.095230 2003 rpcz_store.cc:276] 0112 02:52:39.033651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.033686 (+ 35us) service_pool.cc:225] Handling call
0112 02:52:40.095081 (+1061395us) ts_tablet_manager.cc:1145] Deleting tablet f776e2a6ec704cf9b64483bc7d5b1dff
0112 02:52:40.095090 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.095155 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1061274}
W20250112 02:52:40.095312 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b8daf4ef3593410eae87672402292be6 because the tablet was not found. No further retry: Not found: Tablet not found: b8daf4ef3593410eae87672402292be6
W20250112 02:52:40.095503 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d45fce86e02f46d4a57af718fa2a62c8 because the tablet was not found. No further retry: Not found: Tablet not found: d45fce86e02f46d4a57af718fa2a62c8
I20250112 02:52:40.095590 1173 tablet_bootstrap.cc:654] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:40.095667 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 8c821ea3ca4946c290c17135366e0cac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c821ea3ca4946c290c17135366e0cac already in progress: creating tablet
W20250112 02:52:40.095854 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f234256b343f447dadfc3d1936a824ee because the tablet was not found. No further retry: Not found: Tablet not found: f234256b343f447dadfc3d1936a824ee
W20250112 02:52:40.096343 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f776e2a6ec704cf9b64483bc7d5b1dff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f776e2a6ec704cf9b64483bc7d5b1dff already in progress: creating tablet
I20250112 02:52:40.097280 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:40.097656 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:40.098064 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:40.095446 2003 tablet_replica.cc:331] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:40.098296 2003 raft_consensus.cc:2238] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:40.098434 2003 raft_consensus.cc:2267] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:40.098754 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_5d473cabff2e46e0b531fbce28d9d558 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 1 got 0
I20250112 02:52:40.098919 2003 ts_tablet_manager.cc:1905] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.099154 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:40.099455 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:40.099869 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_a9907ed8bfe14625977ec98b67e9d599 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 2 got 0
I20250112 02:52:40.102627 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 1 got 0
I20250112 02:52:40.103520 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:40.104005 2047 consensus_queue.cc:1035] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:40.105698 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 09d06e2e8c13464db8cf801a292bdc99 (table default.Table_2c52fead633649dd869763e5aa532be7 [id=e3bb489c53454222a79702b798c0c2f1]). Expected version 3 got 2
I20250112 02:52:40.106563 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:40.108237 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:40.108799 2673 raft_consensus.cc:491] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.108947 2673 raft_consensus.cc:513] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.109562 2673 leader_election.cc:290] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:40.109854 1177 tablet_bootstrap.cc:492] T 47a0d939fdb94590a86389f1f4693e9a P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:40.109992 1177 ts_tablet_manager.cc:1397] T 47a0d939fdb94590a86389f1f4693e9a P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20250112 02:52:40.110388 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b81aee21510452881948e41d7aba9be" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:40.110580 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:40.110775 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b81aee21510452881948e41d7aba9be" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:40.110728 1177 raft_consensus.cc:357] T 47a0d939fdb94590a86389f1f4693e9a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:40.110961 1177 raft_consensus.cc:383] T 47a0d939fdb94590a86389f1f4693e9a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.111016 1177 raft_consensus.cc:738] T 47a0d939fdb94590a86389f1f4693e9a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.111261 1177 consensus_queue.cc:260] T 47a0d939fdb94590a86389f1f4693e9a P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:40.111340 854 leader_election.cc:343] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:40.111865 855 leader_election.cc:343] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:40.111968 855 leader_election.cc:304] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:40.112324 2598 raft_consensus.cc:491] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.112437 2598 raft_consensus.cc:513] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.112993 2598 leader_election.cc:290] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:40.113102 2598 raft_consensus.cc:2743] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:40.114084 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9280db29e54d41af1bed2fa5aa5535" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:40.114408 854 leader_election.cc:343] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 3b9280db29e54d41af1bed2fa5aa5535
I20250112 02:52:40.114583 1177 ts_tablet_manager.cc:1428] T 47a0d939fdb94590a86389f1f4693e9a P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250112 02:52:40.114821 1177 tablet_bootstrap.cc:492] T 3e67f0a0c065498f9a644220414415b0 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:40.114888 855 leader_election.cc:343] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:40.114974 855 leader_election.cc:304] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:40.115332 2598 raft_consensus.cc:2743] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:40.116381 1177 tablet_bootstrap.cc:654] T 3e67f0a0c065498f9a644220414415b0 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.117031 1181 tablet_bootstrap.cc:492] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:40.117244 1181 ts_tablet_manager.cc:1397] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.002s
I20250112 02:52:40.117129 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:40.117934 1181 raft_consensus.cc:357] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.118170 1181 raft_consensus.cc:383] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.118265 1181 raft_consensus.cc:738] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.118497 1181 consensus_queue.cc:260] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.118665 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:40.119130 1181 ts_tablet_manager.cc:1428] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:40.119266 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
I20250112 02:52:40.119858 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 183) took 1288 ms. Trace:
I20250112 02:52:40.119940 2562 rpcz_store.cc:276] 0112 02:52:38.831321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.831595 (+ 274us) service_pool.cc:225] Handling call
0112 02:52:40.089428 (+1257833us) ts_tablet_manager.cc:1145] Deleting tablet 1c639a2bcd4740e4be7b2b162004669a
0112 02:52:40.119310 (+ 29882us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.119845 (+ 535us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1257697}
I20250112 02:52:40.120219 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
W20250112 02:52:40.121003 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 1c639a2bcd4740e4be7b2b162004669a because the tablet was not found. No further retry: Not found: Tablet not found: 1c639a2bcd4740e4be7b2b162004669a
I20250112 02:52:40.120564 2445 consensus_queue.cc:1035] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:40.124164 2024 ts_tablet_manager.cc:1918] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:40.124433 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 184) took 1292 ms. Trace:
I20250112 02:52:40.123860 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5cc414d6420c41839621ab858067548b (table default.Table_32f2fd919490410aacdc9ca9c5a51683 [id=268dae304c9143a881c7a0898632a3e4]). Expected version 1 got 0
I20250112 02:52:40.124533 2562 rpcz_store.cc:276] 0112 02:52:38.831866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.832150 (+ 284us) service_pool.cc:225] Handling call
0112 02:52:40.120182 (+1288032us) ts_tablet_manager.cc:1145] Deleting tablet 61593969d2234ce3a6089027b067d922
0112 02:52:40.121334 (+ 1152us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.124419 (+ 3085us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1287901}
I20250112 02:52:40.124712 1181 tablet_bootstrap.cc:492] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
W20250112 02:52:40.125015 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 61593969d2234ce3a6089027b067d922 because the tablet was not found. No further retry: Not found: Tablet not found: 61593969d2234ce3a6089027b067d922
I20250112 02:52:40.125288 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 185) took 1292 ms. Trace:
I20250112 02:52:40.125358 2562 rpcz_store.cc:276] 0112 02:52:38.832415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.832706 (+ 291us) service_pool.cc:225] Handling call
0112 02:52:40.125223 (+1292517us) ts_tablet_manager.cc:1145] Deleting tablet 8ee98f055024429eafaa29bb961cab87
0112 02:52:40.125230 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.125275 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1292344}
I20250112 02:52:40.125214 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
W20250112 02:52:40.125710 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 8ee98f055024429eafaa29bb961cab87 because the tablet was not found. No further retry: Not found: Tablet not found: 8ee98f055024429eafaa29bb961cab87
I20250112 02:52:40.126041 2562 tablet_replica.cc:331] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:40.126293 2562 raft_consensus.cc:2238] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:40.126789 1181 tablet_bootstrap.cc:654] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.128060 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:40.129184 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:40.131143 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:40.126796 2562 raft_consensus.cc:2267] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:40.136164 2562 ts_tablet_manager.cc:1905] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.137854 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:40.139008 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:40.140973 448 catalog_manager.cc:5526] T 763f77385e404c149c16edabe469c299 P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:40.142331 2024 log.cc:1198] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/d4abf59d96414e5faa7cf75ba133832c
I20250112 02:52:40.142776 2024 ts_tablet_manager.cc:1939] T d4abf59d96414e5faa7cf75ba133832c P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:40.143631 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 93) took 5608 ms. Trace:
I20250112 02:52:40.143854 448 catalog_manager.cc:5526] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:40.146459 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:40.147886 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
I20250112 02:52:40.148229 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:40.150254 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet f924459d1150472aafb325a11825db23
I20250112 02:52:40.150441 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet f3771166b77e4f90aaa116ab1a9dc299
I20250112 02:52:40.150856 452 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40610 (request call id 142) took 4210 ms. Trace:
W20250112 02:52:40.151177 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca2165362ee541adac724b63f39b1489 consensus state has no leader
I20250112 02:52:40.151376 450 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 143) took 4207 ms. Trace:
I20250112 02:52:40.151933 453 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 140) took 4223 ms. Trace:
I20250112 02:52:40.148382 1181 tablet_bootstrap.cc:492] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:40.152529 1181 ts_tablet_manager.cc:1397] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20250112 02:52:40.150996 452 rpcz_store.cc:276] 0112 02:52:35.940288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.120760 (+3180472us) service_pool.cc:225] Handling call
0112 02:52:39.121438 (+ 678us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:39.121626 (+ 188us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:52:39.121699 (+ 73us) catalog_manager.cc:2762] Locking tablets
0112 02:52:39.121860 (+ 161us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:52:39.195024 (+ 73164us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:52:39.271590 (+ 76566us) catalog_manager.cc:2810] Committing in-memory state
0112 02:52:40.150174 (+878584us) catalog_manager.cc:2814] Aborting table tasks
0112 02:52:40.150805 (+ 631us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:39.122604 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:39.122751 (+ 147us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:39.122760 (+ 9us) write_op.cc:435] Acquired schema lock
0112 02:52:39.122765 (+ 5us) tablet.cc:659] Decoding operations
0112 02:52:39.122907 (+ 142us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:39.122921 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:39.122928 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:39.123022 (+ 94us) tablet.cc:698] Row locks acquired
0112 02:52:39.123026 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:39.123056 (+ 30us) write_op.cc:270] Start()
0112 02:52:39.123092 (+ 36us) write_op.cc:276] Timestamp: P: 1736650359122938 usec, L: 0
0112 02:52:39.123097 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:39.123254 (+ 157us) log.cc:844] Serialized 756 byte log entry
0112 02:52:39.127662 (+ 4408us) op_driver.cc:464] REPLICATION: finished
0112 02:52:39.127878 (+ 216us) write_op.cc:301] APPLY: starting
0112 02:52:39.127895 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:39.127950 (+ 55us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:39.127983 (+ 33us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:39.128059 (+ 76us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:39.128103 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:39.128114 (+ 11us) write_op.cc:312] APPLY: finished
0112 02:52:39.128194 (+ 80us) log.cc:844] Serialized 29 byte log entry
0112 02:52:39.128262 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.128281 (+ 19us) write_op.cc:454] Released schema lock
0112 02:52:39.128289 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:39.128373 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.128379 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":162,"apply.run_cpu_time_us":485,"apply.run_wall_time_us":485,"num_ops":2,"prepare.queue_time_us":147,"prepare.run_cpu_time_us":750,"prepare.run_wall_time_us":751,"raft.queue_time_us":1067,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":258,"replication_time_us":4538}]]}
I20250112 02:52:40.153455 899 tablet_service.cc:1514] Processing DeleteTablet for tablet f3771166b77e4f90aaa116ab1a9dc299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:39 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:40.153318 1181 raft_consensus.cc:357] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.153723 1181 raft_consensus.cc:383] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.153803 1181 raft_consensus.cc:738] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.153956 895 tablet_service.cc:1514] Processing DeleteTablet for tablet f924459d1150472aafb325a11825db23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:39 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:40.150993 449 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40610 (request call id 144) took 4200 ms. Trace:
I20250112 02:52:40.154048 1181 consensus_queue.cc:260] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.154225 449 rpcz_store.cc:276] 0112 02:52:35.950771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.142946 (+3192175us) service_pool.cc:225] Handling call
0112 02:52:39.143619 (+ 673us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:39.143786 (+ 167us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:52:39.143865 (+ 79us) catalog_manager.cc:2762] Locking tablets
0112 02:52:39.144087 (+ 222us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:52:39.274548 (+130461us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:52:39.274661 (+ 113us) catalog_manager.cc:2810] Committing in-memory state
0112 02:52:40.150363 (+875702us) catalog_manager.cc:2814] Aborting table tasks
0112 02:52:40.150933 (+ 570us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:39.144888 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:39.144995 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:39.145002 (+ 7us) write_op.cc:435] Acquired schema lock
0112 02:52:39.145005 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:39.145117 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:39.145128 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:39.145135 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:39.145246 (+ 111us) tablet.cc:698] Row locks acquired
0112 02:52:39.145250 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:39.145279 (+ 29us) write_op.cc:270] Start()
0112 02:52:39.145308 (+ 29us) write_op.cc:276] Timestamp: P: 1736650359145260 usec, L: 0
0112 02:52:39.145313 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:39.145465 (+ 152us) log.cc:844] Serialized 756 byte log entry
0112 02:52:39.268478 (+123013us) op_driver.cc:464] REPLICATION: finished
0112 02:52:39.269366 (+ 888us) write_op.cc:301] APPLY: starting
0112 02:52:39.269379 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:39.269419 (+ 40us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:39.269451 (+ 32us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:39.269515 (+ 64us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:39.269547 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:39.269557 (+ 10us) write_op.cc:312] APPLY: finished
0112 02:52:39.269623 (+ 66us) log.cc:844] Serialized 29 byte log entry
0112 02:52:39.269664 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.269679 (+ 15us) write_op.cc:454] Released schema lock
0112 02:52:39.269686 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:39.269753 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.269759 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":853,"apply.run_cpu_time_us":378,"apply.run_wall_time_us":377,"num_ops":2,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":650,"prepare.run_wall_time_us":649,"replication_time_us":123147}]]}
I20250112 02:52:40.154670 1181 ts_tablet_manager.cc:1428] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:40.154855 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:40.154902 1181 tablet_bootstrap.cc:492] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:40.151490 450 rpcz_store.cc:276] 0112 02:52:35.943834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.136703 (+3192869us) service_pool.cc:225] Handling call
0112 02:52:39.138177 (+ 1474us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:39.138416 (+ 239us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:39.138592 (+ 176us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:39.138659 (+ 67us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:39.273026 (+134367us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:40.151303 (+878277us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:39.139195 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:39.139304 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:39.139311 (+ 7us) write_op.cc:435] Acquired schema lock
0112 02:52:39.139315 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:39.139417 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:39.139430 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:39.139436 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:39.139525 (+ 89us) tablet.cc:698] Row locks acquired
0112 02:52:39.139529 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:39.139566 (+ 37us) write_op.cc:270] Start()
0112 02:52:39.139595 (+ 29us) write_op.cc:276] Timestamp: P: 1736650359139547 usec, L: 0
0112 02:52:39.139601 (+ 6us) op_driver.cc:348] REPLICATION: starting
0112 02:52:39.139769 (+ 168us) log.cc:844] Serialized 405 byte log entry
0112 02:52:39.268414 (+128645us) op_driver.cc:464] REPLICATION: finished
0112 02:52:39.268734 (+ 320us) write_op.cc:301] APPLY: starting
0112 02:52:39.268754 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:39.268806 (+ 52us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:39.268876 (+ 70us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:39.268961 (+ 85us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:39.269006 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:39.269018 (+ 12us) write_op.cc:312] APPLY: finished
0112 02:52:39.269117 (+ 99us) log.cc:844] Serialized 23 byte log entry
0112 02:52:39.269184 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.269203 (+ 19us) write_op.cc:454] Released schema lock
0112 02:52:39.269210 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:39.269290 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.269296 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":236,"apply.run_cpu_time_us":549,"apply.run_wall_time_us":563,"num_ops":1,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":650,"prepare.run_wall_time_us":650,"replication_time_us":128786}]]}
I20250112 02:52:40.156092 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 6913b0ac9ceb4dbc9cdf591f159134f7 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:40.156440 450 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:40610 (request call id 149) took 3814 ms. Trace:
I20250112 02:52:40.156522 450 rpcz_store.cc:276] 0112 02:52:36.342494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.156008 (+3813514us) service_pool.cc:225] Handling call
0112 02:52:40.156410 (+ 402us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:52:40.156697 450 master_service.cc:946] ReplaceTablet: received request to replace tablet bab4978edea0445fb0456e88e7fb4c44 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:40.156947 1181 tablet_bootstrap.cc:654] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.152061 453 rpcz_store.cc:276] 0112 02:52:35.928057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.087765 (+3159708us) service_pool.cc:225] Handling call
0112 02:52:39.088250 (+ 485us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:39.088457 (+ 207us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:39.088616 (+ 159us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:39.088672 (+ 56us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:39.206601 (+117929us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:40.151892 (+945291us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:39.089422 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:39.089510 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:39.089517 (+ 7us) write_op.cc:435] Acquired schema lock
0112 02:52:39.089519 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:39.089633 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:39.089647 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:39.089655 (+ 8us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:39.089731 (+ 76us) tablet.cc:698] Row locks acquired
0112 02:52:39.089735 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:39.089777 (+ 42us) write_op.cc:270] Start()
0112 02:52:39.089823 (+ 46us) write_op.cc:276] Timestamp: P: 1736650359089656 usec, L: 0
0112 02:52:39.089827 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:39.089988 (+ 161us) log.cc:844] Serialized 405 byte log entry
0112 02:52:39.093437 (+ 3449us) op_driver.cc:464] REPLICATION: finished
0112 02:52:39.093542 (+ 105us) write_op.cc:301] APPLY: starting
0112 02:52:39.093556 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:39.093594 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:39.093627 (+ 33us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:39.093673 (+ 46us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:39.093703 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:39.093710 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:39.093789 (+ 79us) log.cc:844] Serialized 23 byte log entry
0112 02:52:39.093842 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.093859 (+ 17us) write_op.cc:454] Released schema lock
0112 02:52:39.093864 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:39.093936 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.093940 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2908,"child_traces":[["op",{"apply.queue_time_us":63,"apply.run_cpu_time_us":384,"apply.run_wall_time_us":382,"num_ops":1,"prepare.queue_time_us":232,"prepare.run_cpu_time_us":664,"prepare.run_wall_time_us":664,"raft.queue_time_us":386,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":195,"replication_time_us":3566}]]}
I20250112 02:52:40.157526 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_82731e14956e49edb4d2103794e88fec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:40.138451 2609 raft_consensus.cc:491] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.157789 2609 raft_consensus.cc:513] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:40.158056 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82731e14956e49edb4d2103794e88fec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:40.158427 2609 leader_election.cc:290] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.138780 1173 tablet_bootstrap.cc:492] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:40.137531 2208 raft_consensus.cc:491] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.158650 1173 ts_tablet_manager.cc:1397] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.065s user 0.000s sys 0.003s
I20250112 02:52:40.158625 2208 raft_consensus.cc:513] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.159204 2208 leader_election.cc:290] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.152166 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet f924459d1150472aafb325a11825db23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:39 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:40.159406 1173 raft_consensus.cc:357] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.152305 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet f3771166b77e4f90aaa116ab1a9dc299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:39 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:40.159601 1173 raft_consensus.cc:383] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.159675 1173 raft_consensus.cc:738] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.159875 1173 consensus_queue.cc:260] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.160050 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e67f0a0c065498f9a644220414415b0" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:40.160189 917 raft_consensus.cc:2463] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:40.160389 1173 ts_tablet_manager.cc:1428] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:40.160548 989 leader_election.cc:304] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:40.160671 1173 tablet_bootstrap.cc:492] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:40.160938 2208 raft_consensus.cc:2798] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:40.161037 2208 raft_consensus.cc:491] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:40.161123 2208 raft_consensus.cc:3054] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:40.162142 1173 tablet_bootstrap.cc:654] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.163828 2208 raft_consensus.cc:513] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:40.164417 989 leader_election.cc:343] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 352c87f1dc724e7b94bc2e64abfce8d7
I20250112 02:52:40.165019 2208 leader_election.cc:290] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.167549 2024 rpcz_store.cc:276] 0112 02:52:34.535288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.540760 (+ 5472us) service_pool.cc:225] Handling call
0112 02:52:40.065072 (+5524312us) ts_tablet_manager.cc:1145] Deleting tablet d4abf59d96414e5faa7cf75ba133832c
0112 02:52:40.065081 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.069935 (+ 4854us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.124046 (+ 54111us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.143581 (+ 19535us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5522073}
I20250112 02:52:40.168431 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet d4abf59d96414e5faa7cf75ba133832c (table default.Table_525750dad2b7461586d070dbaa74d0b1 [id=7d048aa907a74605964b68726ac5d759]) successfully deleted
I20250112 02:52:40.168934 2024 tablet_replica.cc:331] T e5c10fad790046a88fbc8ebe0303c481 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:40.169530 2024 raft_consensus.cc:2238] T e5c10fad790046a88fbc8ebe0303c481 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:40.170039 2024 raft_consensus.cc:2267] T e5c10fad790046a88fbc8ebe0303c481 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:40.172533 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e67f0a0c065498f9a644220414415b0" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:40.172714 918 raft_consensus.cc:3054] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:40.173707 2024 ts_tablet_manager.cc:1905] T e5c10fad790046a88fbc8ebe0303c481 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.174975 918 raft_consensus.cc:2463] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:40.175407 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_2c52fead633649dd869763e5aa532be7 [id=e3bb489c53454222a79702b798c0c2f1] alter complete (version 3)
W20250112 02:52:40.175453 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17c54c5dc86446cf85974777c8145ef6 consensus state has no leader
I20250112 02:52:40.175863 989 leader_election.cc:304] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:40.176512 2208 raft_consensus.cc:2798] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:40.176743 2208 raft_consensus.cc:695] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:40.177039 2208 consensus_queue.cc:237] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.179257 2003 ts_tablet_manager.cc:1918] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:40.179366 2003 log.cc:1198] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/4348fe4c32704bf7b3a8d21ff3b8ac1b
I20250112 02:52:40.179656 2003 ts_tablet_manager.cc:1939] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:40.179936 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 248) took 1146 ms. Trace:
I20250112 02:52:40.180056 2003 rpcz_store.cc:276] 0112 02:52:39.033580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.033869 (+ 289us) service_pool.cc:225] Handling call
0112 02:52:40.095412 (+1061543us) ts_tablet_manager.cc:1145] Deleting tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b
0112 02:52:40.095419 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.130081 (+ 34662us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.179151 (+ 49070us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.179921 (+ 770us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1061441}
I20250112 02:52:40.180477 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 247) took 1147 ms. Trace:
I20250112 02:52:40.181082 447 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 145) took 4230 ms. Trace:
I20250112 02:52:40.181181 447 rpcz_store.cc:276] 0112 02:52:35.950871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.147945 (+3197074us) service_pool.cc:225] Handling call
0112 02:52:39.148433 (+ 488us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:39.148703 (+ 270us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:39.148865 (+ 162us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:39.149004 (+ 139us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:39.279225 (+130221us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:40.181045 (+901820us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:39.149413 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:39.149487 (+ 74us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:39.149492 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:39.149494 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:39.149565 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:39.149573 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:39.149577 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:39.149637 (+ 60us) tablet.cc:698] Row locks acquired
0112 02:52:39.149642 (+ 5us) write_op.cc:260] PREPARE: finished
0112 02:52:39.149667 (+ 25us) write_op.cc:270] Start()
0112 02:52:39.149690 (+ 23us) write_op.cc:276] Timestamp: P: 1736650359149648 usec, L: 0
0112 02:52:39.149693 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:39.149828 (+ 135us) log.cc:844] Serialized 405 byte log entry
0112 02:52:39.268524 (+118696us) op_driver.cc:464] REPLICATION: finished
0112 02:52:39.269819 (+ 1295us) write_op.cc:301] APPLY: starting
0112 02:52:39.269831 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:39.269861 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:39.269885 (+ 24us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:39.269917 (+ 32us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:39.269947 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:39.269958 (+ 11us) write_op.cc:312] APPLY: finished
0112 02:52:39.270006 (+ 48us) log.cc:844] Serialized 23 byte log entry
0112 02:52:39.270064 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.270077 (+ 13us) write_op.cc:454] Released schema lock
0112 02:52:39.270084 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":119,"child_traces":[["op",{"apply.queue_time_us":1265,"apply.run_cpu_time_us":319,"apply.run_wall_time_us":319,"num_ops":1,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":490,"prepare.run_wall_time_us":490,"replication_time_us":118816,"thread_start_us":25744,"threads_started":1}]]}
I20250112 02:52:40.181752 447 master_service.cc:946] ReplaceTablet: received request to replace tablet b61f16f3d601436bb4b32344de28c1a7 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:40.182127 447 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:40610 (request call id 152) took 3586 ms. Trace:
I20250112 02:52:40.182215 447 rpcz_store.cc:276] 0112 02:52:36.595788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.181694 (+3585906us) service_pool.cc:225] Handling call
0112 02:52:40.182078 (+ 384us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:52:40.182574 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_56415126d9a642c386bd4058b011e2b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:40.183115 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56415126d9a642c386bd4058b011e2b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:40.186877 2003 rpcz_store.cc:276] 0112 02:52:39.033468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.034019 (+ 551us) service_pool.cc:225] Handling call
0112 02:52:40.180370 (+1146351us) ts_tablet_manager.cc:1145] Deleting tablet b1d9fae290ec4281839991f9d40b105c
0112 02:52:40.180380 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.180462 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1146238}
I20250112 02:52:40.187258 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 255) took 1071 ms. Trace:
I20250112 02:52:40.187361 2003 rpcz_store.cc:276] 0112 02:52:39.115985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.116722 (+ 737us) service_pool.cc:225] Handling call
0112 02:52:40.187158 (+1070436us) ts_tablet_manager.cc:1145] Deleting tablet 058a9ca089cd4967887b0a1127c7d03f
0112 02:52:40.187168 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.187245 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1051692}
I20250112 02:52:40.187551 2673 raft_consensus.cc:491] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.187824 2003 tablet_replica.cc:331] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:40.187716 2673 raft_consensus.cc:513] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.188437 2673 leader_election.cc:290] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:40.188797 2003 raft_consensus.cc:2238] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:40.192634 450 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:40610 (request call id 150) took 3830 ms. Trace:
I20250112 02:52:40.192757 450 rpcz_store.cc:276] 0112 02:52:36.362602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.156648 (+3794046us) service_pool.cc:225] Handling call
0112 02:52:40.192564 (+ 35916us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:52:40.193240 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_0182c19fa6214ea1928001cb0359f262"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:40.193817 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0182c19fa6214ea1928001cb0359f262 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:40.195834 2673 raft_consensus.cc:491] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.195955 2673 raft_consensus.cc:513] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.196444 2673 leader_election.cc:290] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:40.200300 2003 raft_consensus.cc:2267] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:40.200875 991 leader_election.cc:343] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: 352c87f1dc724e7b94bc2e64abfce8d7
I20250112 02:52:40.201041 991 leader_election.cc:304] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:40.201253 453 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 151) took 3633 ms. Trace:
I20250112 02:52:40.201529 2609 raft_consensus.cc:2743] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:52:40.201395 453 rpcz_store.cc:276] 0112 02:52:36.567836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.157302 (+3589466us) service_pool.cc:225] Handling call
0112 02:52:40.180922 (+ 23620us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:40.181370 (+ 448us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:40.200979 (+ 19609us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:40.201113 (+ 134us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:40.201210 (+ 97us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:40.184747 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.184817 (+ 70us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.184833 (+ 16us) write_op.cc:435] Acquired schema lock
0112 02:52:40.184835 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:40.184902 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.184908 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.184912 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:40.184969 (+ 57us) tablet.cc:698] Row locks acquired
0112 02:52:40.184987 (+ 18us) write_op.cc:260] PREPARE: finished
0112 02:52:40.185010 (+ 23us) write_op.cc:270] Start()
0112 02:52:40.185041 (+ 31us) write_op.cc:276] Timestamp: P: 1736650360184991 usec, L: 0
0112 02:52:40.185044 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.185164 (+ 120us) log.cc:844] Serialized 433 byte log entry
0112 02:52:40.199584 (+ 14420us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.200362 (+ 778us) write_op.cc:301] APPLY: starting
0112 02:52:40.200374 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.200404 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.200440 (+ 36us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.200583 (+ 143us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.200621 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:40.200632 (+ 11us) write_op.cc:312] APPLY: finished
0112 02:52:40.200695 (+ 63us) log.cc:844] Serialized 29 byte log entry
0112 02:52:40.200739 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.200756 (+ 17us) write_op.cc:454] Released schema lock
0112 02:52:40.200764 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":749,"apply.run_cpu_time_us":464,"apply.run_wall_time_us":463,"num_ops":2,"prepare.queue_time_us":3047,"prepare.run_cpu_time_us":457,"prepare.run_wall_time_us":456,"replication_time_us":14527,"threads_started":1}]]}
I20250112 02:52:40.201947 747 tablet_service.cc:1514] Processing DeleteTablet for tablet f924459d1150472aafb325a11825db23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:39 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.202541 2003 ts_tablet_manager.cc:1905] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.201982 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e67f0a0c065498f9a644220414415b0" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:40.203385 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e67f0a0c065498f9a644220414415b0" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:40.204033 991 leader_election.cc:343] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:40.205960 744 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ffdc237f8640f0b378809f93fc6292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.206117 749 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d411f906b4d4eb68343bdcc493e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:40.206334 854 leader_election.cc:343] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 896908b2726c40f9aabdfc9a6b3283f7
W20250112 02:52:40.206832 989 consensus_peers.cc:487] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4a3486ef011492c8a1df8d6278adedf. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:40.203706 747 tablet_service.cc:1514] Processing DeleteTablet for tablet f3771166b77e4f90aaa116ab1a9dc299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:39 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.207001 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1954e7859f422f9690fe4a9384a8bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:40.207077 854 leader_election.cc:343] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 05000c6a48e0450d8e6685ca864f67f0
W20250112 02:52:40.207417 991 leader_election.cc:343] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:40.208397 756 tablet_service.cc:1514] Processing DeleteTablet for tablet bab4978edea0445fb0456e88e7fb4c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.208586 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c821ea3ca4946c290c17135366e0cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.208719 743 tablet_service.cc:1514] Processing DeleteTablet for tablet f234256b343f447dadfc3d1936a824ee with delete_type TABLET_DATA_DELETED (replaced by tablet 82b351c563d749df806ab5d0331a4ef2) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.208833 751 tablet_service.cc:1514] Processing DeleteTablet for tablet b9c006281dc949119951c74e23167d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.208969 754 tablet_service.cc:1514] Processing DeleteTablet for tablet f05c8426ea3d4fe89fbe6c88c688019f with delete_type TABLET_DATA_DELETED (replaced by tablet c5079f87652a4ea58b366d1f9448fd4e) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.209066 743 tablet_service.cc:1514] Processing DeleteTablet for tablet c7c530aa0aa64e47ae27d61f341d2bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.209157 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ee98f055024429eafaa29bb961cab87 with delete_type TABLET_DATA_DELETED (replaced by tablet 99407ead9cf547e3bbe78c0d0b327e6a) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.209241 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b351c563d749df806ab5d0331a4ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.209335 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f4bd2df0ce45dabc55373f1154d3ac with delete_type TABLET_DATA_DELETED (replaced by tablet f559dedfb7854cf890bef362c50b4213) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.209424 749 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e7882e07784bdda18dc1ffd20963c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.209509 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 06edfca60eed4dd3a2dee7b662b0970c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.209597 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a34d03a80ae402db6607261a5c40b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.209697 743 tablet_service.cc:1514] Processing DeleteTablet for tablet cbe30f9860e2492b90e8cedf8ff9daba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.209867 751 tablet_service.cc:1514] Processing DeleteTablet for tablet a12c5d58e6c246c983e8a533906ce77d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.206615 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896908b2726c40f9aabdfc9a6b3283f7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:40.209995 2562 ts_tablet_manager.cc:1918] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:40.210088 2562 log.cc:1198] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/b6d84027ad6d40a5be5afcdd854431a5
I20250112 02:52:40.210109 746 tablet_service.cc:1514] Processing DeleteTablet for tablet c29a9ca43e034b43a00207e5847bbee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.210150 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 54215c55966a46e1ab96e352c4e3764e with delete_type TABLET_DATA_DELETED (replaced by tablet a470f84d50df4cf0b4a1e5d248b4c32a) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.210302 756 tablet_service.cc:1514] Processing DeleteTablet for tablet e5c10fad790046a88fbc8ebe0303c481 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.210417 2562 ts_tablet_manager.cc:1939] T b6d84027ad6d40a5be5afcdd854431a5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:40.210433 744 tablet_service.cc:1514] Processing DeleteTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.210546 743 tablet_service.cc:1514] Processing DeleteTablet for tablet fe614eaeb793473d9b2da206b7345562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.210825 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05000c6a48e0450d8e6685ca864f67f0" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:40.210747 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 186) took 1377 ms. Trace:
I20250112 02:52:40.211086 2562 rpcz_store.cc:276] 0112 02:52:38.833011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.833305 (+ 294us) service_pool.cc:225] Handling call
0112 02:52:40.125992 (+1292687us) ts_tablet_manager.cc:1145] Deleting tablet b6d84027ad6d40a5be5afcdd854431a5
0112 02:52:40.126001 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.140523 (+ 14522us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.209913 (+ 69390us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.210733 (+ 820us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1292567}
I20250112 02:52:40.211376 2562 tablet_replica.cc:331] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:40.211489 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet b6d84027ad6d40a5be5afcdd854431a5 (table default.Table_31424b670f274fd3abd17eb8451ba95c [id=ebc9aae273fb4bcd8bd904f2179f1dd4]) successfully deleted
I20250112 02:52:40.210072 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 13578542e9e741e8817e0ee63b10f1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.213092 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 70715db0394d46d9b6b289160b00be7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.213266 743 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a3486ef011492c8a1df8d6278adedf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.209988 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c04f1a2cb04950a9f6170e252c19b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.213397 747 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5b3c712c0f448ea9b14ab93dd731ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.213438 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 87759e8b99b04e628badc8f60cb38bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.213809 2609 raft_consensus.cc:491] T 4fc8658b0efe43bdb339edbff64973f6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.213953 2609 raft_consensus.cc:513] T 4fc8658b0efe43bdb339edbff64973f6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.214182 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 416fb6820cd54e118950c68d381f51b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.214583 2609 leader_election.cc:290] T 4fc8658b0efe43bdb339edbff64973f6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.214762 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b (table default.Table_1d107b48a09e41d6bb3401b01a21ee78 [id=68bab352836142c39c109e8e02553802]) successfully deleted
I20250112 02:52:40.214900 749 tablet_service.cc:1514] Processing DeleteTablet for tablet d659c5a4f7a446c6aaf2283f65b45106 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:40.215018 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b1d9fae290ec4281839991f9d40b105c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1d9fae290ec4281839991f9d40b105c already in progress: creating tablet
I20250112 02:52:40.215090 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 416fb6820cd54e118950c68d381f51b2) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.215123 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc8658b0efe43bdb339edbff64973f6" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:40.215236 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 058a9ca089cd4967887b0a1127c7d03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 058a9ca089cd4967887b0a1127c7d03f already in progress: creating tablet
I20250112 02:52:40.215318 2445 raft_consensus.cc:491] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.215409 2445 raft_consensus.cc:513] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.216006 2445 leader_election.cc:290] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.217124 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd6d31155034908ac44a90c3a2ea7d6" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:40.217346 1050 raft_consensus.cc:2463] T afd6d31155034908ac44a90c3a2ea7d6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:40.217839 714 leader_election.cc:304] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:40.218106 2445 raft_consensus.cc:2798] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:40.218165 2445 raft_consensus.cc:491] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:40.218218 2445 raft_consensus.cc:3054] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:40.211650 855 leader_election.cc:343] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:40.220641 2445 raft_consensus.cc:513] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.220727 855 leader_election.cc:304] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:40.221107 855 leader_election.cc:343] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:40.221186 855 leader_election.cc:304] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:40.221444 2598 raft_consensus.cc:2743] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:40.215678 2673 raft_consensus.cc:491] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.221513 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd6d31155034908ac44a90c3a2ea7d6" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:40.221676 910 raft_consensus.cc:3054] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:40.221657 2673 raft_consensus.cc:513] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.222244 2673 leader_election.cc:290] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:40.216147 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc8658b0efe43bdb339edbff64973f6" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:40.222434 918 raft_consensus.cc:2388] T 4fc8658b0efe43bdb339edbff64973f6 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1e775296eb746db8b318f17d26820e6 in current term 1: Already voted for candidate 289afc116aa74408aa5bff74e11c50d8 in this term.
W20250112 02:52:40.222822 854 leader_election.cc:343] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 24ebf85f76f44633a20d97964638c0ac
I20250112 02:52:40.222985 895 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5b3c712c0f448ea9b14ab93dd731ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:40.223301 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f4bd2df0ce45dabc55373f1154d3ac with delete_type TABLET_DATA_DELETED (replaced by tablet f559dedfb7854cf890bef362c50b4213) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:40.223563 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1954e7859f422f9690fe4a9384a8bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:40.223649 989 leader_election.cc:304] T 4fc8658b0efe43bdb339edbff64973f6 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:40.223718 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 54215c55966a46e1ab96e352c4e3764e with delete_type TABLET_DATA_DELETED (replaced by tablet a470f84d50df4cf0b4a1e5d248b4c32a) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:40.223868 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b351c563d749df806ab5d0331a4ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:40.224011 2208 raft_consensus.cc:3054] T 4fc8658b0efe43bdb339edbff64973f6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:40.224486 910 raft_consensus.cc:2463] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:40.224539 897 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ffdc237f8640f0b378809f93fc6292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:40.224750 2562 raft_consensus.cc:2238] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:40.224834 713 leader_election.cc:304] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:40.225028 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 13578542e9e741e8817e0ee63b10f1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:40.225070 2047 raft_consensus.cc:2798] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:40.225174 2047 raft_consensus.cc:695] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:40.225250 2562 raft_consensus.cc:2267] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:40.225425 2047 consensus_queue.cc:237] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.225703 2562 ts_tablet_manager.cc:1905] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.216531 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd6d31155034908ac44a90c3a2ea7d6" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:40.226749 917 raft_consensus.cc:2371] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1. Re-sending same reply.
I20250112 02:52:40.221575 2696 raft_consensus.cc:2743] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:40.227912 2024 ts_tablet_manager.cc:1918] T e5c10fad790046a88fbc8ebe0303c481 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:40.227998 2024 log.cc:1198] T e5c10fad790046a88fbc8ebe0303c481 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/e5c10fad790046a88fbc8ebe0303c481
I20250112 02:52:40.228339 2024 ts_tablet_manager.cc:1939] T e5c10fad790046a88fbc8ebe0303c481 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:40.228577 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 85) took 5694 ms. Trace:
I20250112 02:52:40.228631 2024 rpcz_store.cc:276] 0112 02:52:34.534480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.534537 (+ 57us) service_pool.cc:225] Handling call
0112 02:52:40.168815 (+5634278us) ts_tablet_manager.cc:1145] Deleting tablet e5c10fad790046a88fbc8ebe0303c481
0112 02:52:40.168829 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.197956 (+ 29127us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.227830 (+ 29874us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.228566 (+ 736us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5625798}
I20250112 02:52:40.229053 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet e5c10fad790046a88fbc8ebe0303c481 (table default.Table_16b036d8833a47a99577ab67579ccb5a [id=eb980088c4584ccf82bb585c2210894a]) successfully deleted
I20250112 02:52:40.229306 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24ebf85f76f44633a20d97964638c0ac" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:40.229743 897 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d411f906b4d4eb68343bdcc493e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:38786
W20250112 02:52:40.229920 855 leader_election.cc:343] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:40.230031 855 leader_election.cc:304] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:40.230484 897 tablet_service.cc:1514] Processing DeleteTablet for tablet cbe30f9860e2492b90e8cedf8ff9daba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:38786
W20250112 02:52:40.232074 991 consensus_peers.cc:487] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 416fb6820cd54e118950c68d381f51b2. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:40.235937 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c04f1a2cb04950a9f6170e252c19b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:40.236065 747 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d84027ad6d40a5be5afcdd854431a5 with delete_type TABLET_DATA_DELETED (replaced by tablet fbe9e9624ede40c29ea852b2b5494965) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.236358 747 tablet_service.cc:1514] Processing DeleteTablet for tablet a85300a9aa1c4c50a3957a004bea7933 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.236672 897 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d84027ad6d40a5be5afcdd854431a5 with delete_type TABLET_DATA_DELETED (replaced by tablet fbe9e9624ede40c29ea852b2b5494965) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:40.237102 897 tablet_service.cc:1514] Processing DeleteTablet for tablet c29a9ca43e034b43a00207e5847bbee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:40.237095 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd6d31155034908ac44a90c3a2ea7d6" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:40.237236 1050 raft_consensus.cc:3054] T afd6d31155034908ac44a90c3a2ea7d6 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:40.237488 897 tablet_service.cc:1514] Processing DeleteTablet for tablet bab4978edea0445fb0456e88e7fb4c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:40.238023 897 tablet_service.cc:1514] Processing DeleteTablet for tablet b1d9fae290ec4281839991f9d40b105c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:40.238298 897 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e7882e07784bdda18dc1ffd20963c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
W20250112 02:52:40.239162 455 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:58959 (request call id 240) took 1926 ms (client timeout 1000 ms). Trace:
W20250112 02:52:40.239267 455 rpcz_store.cc:269] 0112 02:52:38.312603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.002545 (+689942us) service_pool.cc:225] Handling call
0112 02:52:40.239130 (+1236585us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:39.541318 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:39.541426 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:39.541432 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:39.541434 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:39.541531 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:39.541545 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:39.541552 (+ 7us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:39.541618 (+ 66us) tablet.cc:698] Row locks acquired
0112 02:52:39.541620 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:39.541679 (+ 59us) write_op.cc:270] Start()
0112 02:52:39.541712 (+ 33us) write_op.cc:276] Timestamp: P: 1736650359541555 usec, L: 0
0112 02:52:39.541715 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:39.541884 (+ 169us) log.cc:844] Serialized 527 byte log entry
0112 02:52:39.544871 (+ 2987us) op_driver.cc:464] REPLICATION: finished
0112 02:52:39.544997 (+ 126us) write_op.cc:301] APPLY: starting
0112 02:52:39.545009 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:39.545062 (+ 53us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:39.545116 (+ 54us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:39.545180 (+ 64us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:39.545208 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:39.545215 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:39.545324 (+ 109us) log.cc:844] Serialized 23 byte log entry
0112 02:52:39.545384 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.545400 (+ 16us) write_op.cc:454] Released schema lock
0112 02:52:39.545405 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:39.545469 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.545472 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":113522,"reported_tablets":175,"child_traces":[["op",{"apply.queue_time_us":80,"apply.run_cpu_time_us":464,"apply.run_wall_time_us":465,"num_ops":1,"prepare.queue_time_us":102,"prepare.run_cpu_time_us":676,"prepare.run_wall_time_us":677,"raft.queue_time_us":208,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":297,"replication_time_us":3132,"spinlock_wait_cycles":121216}]]}
I20250112 02:52:40.240069 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 352c87f1dc724e7b94bc2e64abfce8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.242027 1050 raft_consensus.cc:2463] T afd6d31155034908ac44a90c3a2ea7d6 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:40.242395 2445 leader_election.cc:290] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.242575 1173 tablet_bootstrap.cc:492] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:40.242786 2673 raft_consensus.cc:2743] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:40.244529 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b184806b4884cd3b0a002905a1e57af with delete_type TABLET_DATA_DELETED (replaced by tablet 47a0d939fdb94590a86389f1f4693e9a) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.245223 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dda52f012794493a102ed1abc718f5b with delete_type TABLET_DATA_DELETED (replaced by tablet 17c54c5dc86446cf85974777c8145ef6) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.245922 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af55dab202444358ea1a0674d37946d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.246277 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 73136be25c79491784d3ea71aa9b27f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.246416 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb76746b3a846c3aba531c04bf63966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.242722 1173 ts_tablet_manager.cc:1397] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.082s user 0.004s sys 0.008s
I20250112 02:52:40.249330 1173 raft_consensus.cc:357] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.249517 1173 raft_consensus.cc:383] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.249583 1173 raft_consensus.cc:738] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.249778 1173 consensus_queue.cc:260] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.250337 1173 ts_tablet_manager.cc:1428] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:40.250607 1173 tablet_bootstrap.cc:492] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:40.252430 749 tablet_service.cc:1514] Processing DeleteTablet for tablet d4abf59d96414e5faa7cf75ba133832c with delete_type TABLET_DATA_DELETED (replaced by tablet 5a3af0ea4e954053ae89814f3e74e6b7) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.252811 749 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a7281c0c09442ebd0cc13b8496c86e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.253149 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 058a9ca089cd4967887b0a1127c7d03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.254104 749 tablet_service.cc:1514] Processing DeleteTablet for tablet d45fce86e02f46d4a57af718fa2a62c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 5804fc4130f84c69832c1022b7bd7fa5) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:40.254494 991 consensus_peers.cc:487] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4a3486ef011492c8a1df8d6278adedf. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:40.254956 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a18132549374218893f35a4e4226b72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.255614 2024 tablet_replica.cc:331] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:40.259393 749 tablet_service.cc:1514] Processing DeleteTablet for tablet b8daf4ef3593410eae87672402292be6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.259778 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 003a3b4705e943c6a900762ec11ddd47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.260200 1181 tablet_bootstrap.cc:492] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:40.260322 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 2730634a5cfd4c2fb5c4e30100c556d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.260353 1181 ts_tablet_manager.cc:1397] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.106s user 0.000s sys 0.007s
I20250112 02:52:40.260697 749 tablet_service.cc:1514] Processing DeleteTablet for tablet b61f16f3d601436bb4b32344de28c1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.261034 1181 raft_consensus.cc:357] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.261229 1181 raft_consensus.cc:383] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.261294 1181 raft_consensus.cc:738] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.261493 1181 consensus_queue.cc:260] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.262285 1181 ts_tablet_manager.cc:1428] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20250112 02:52:40.267113 454 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:37987 (request call id 254) took 2386 ms (client timeout 1000 ms). Trace:
I20250112 02:52:40.267432 1181 tablet_bootstrap.cc:492] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:40.268949 1181 tablet_bootstrap.cc:654] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:40.267271 454 rpcz_store.cc:269] 0112 02:52:37.880684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.263450 (+382766us) service_pool.cc:225] Handling call
0112 02:52:40.267075 (+2003625us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:38.781743 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:38.781873 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:38.781881 (+ 8us) write_op.cc:435] Acquired schema lock
0112 02:52:38.781885 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:38.782015 (+ 130us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:38.782029 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:38.782036 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:38.782162 (+ 126us) tablet.cc:698] Row locks acquired
0112 02:52:38.782166 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:38.782201 (+ 35us) write_op.cc:270] Start()
0112 02:52:38.782235 (+ 34us) write_op.cc:276] Timestamp: P: 1736650358782149 usec, L: 0
0112 02:52:38.782241 (+ 6us) op_driver.cc:348] REPLICATION: starting
0112 02:52:38.782427 (+ 186us) log.cc:844] Serialized 901 byte log entry
0112 02:52:38.785119 (+ 2692us) op_driver.cc:464] REPLICATION: finished
0112 02:52:38.785223 (+ 104us) write_op.cc:301] APPLY: starting
0112 02:52:38.785234 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:38.785267 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:38.785285 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:38.785333 (+ 48us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:38.785359 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:38.785365 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:38.785412 (+ 47us) log.cc:844] Serialized 29 byte log entry
0112 02:52:38.785453 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.785467 (+ 14us) write_op.cc:454] Released schema lock
0112 02:52:38.785472 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:38.785525 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:38.785528 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":26686,"reported_tablets":205,"child_traces":[["op",{"apply.queue_time_us":69,"apply.run_cpu_time_us":295,"apply.run_wall_time_us":294,"num_ops":2,"prepare.queue_time_us":9521,"prepare.run_cpu_time_us":803,"prepare.run_wall_time_us":803,"raft.queue_time_us":272,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":209,"replication_time_us":2844}]]}
I20250112 02:52:40.271605 1177 tablet_bootstrap.cc:492] T 3e67f0a0c065498f9a644220414415b0 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:40.271924 2024 raft_consensus.cc:2238] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:40.272310 2024 raft_consensus.cc:2267] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:40.272845 2024 ts_tablet_manager.cc:1905] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.256705 1173 tablet_bootstrap.cc:654] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.277684 2208 raft_consensus.cc:2743] T 4fc8658b0efe43bdb339edbff64973f6 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:40.283253 1181 tablet_bootstrap.cc:492] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:40.283406 1181 ts_tablet_manager.cc:1397] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20250112 02:52:40.284164 1181 raft_consensus.cc:357] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:40.286989 1181 raft_consensus.cc:383] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.287078 1181 raft_consensus.cc:738] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.272382 1177 ts_tablet_manager.cc:1397] T 3e67f0a0c065498f9a644220414415b0 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.158s user 0.007s sys 0.004s
I20250112 02:52:40.287282 1181 consensus_queue.cc:260] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:40.288161 1177 raft_consensus.cc:357] T 3e67f0a0c065498f9a644220414415b0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.288419 1181 ts_tablet_manager.cc:1428] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250112 02:52:40.288502 1177 raft_consensus.cc:383] T 3e67f0a0c065498f9a644220414415b0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.288739 1177 raft_consensus.cc:738] T 3e67f0a0c065498f9a644220414415b0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.289199 1177 consensus_queue.cc:260] T 3e67f0a0c065498f9a644220414415b0 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:40.289955 991 consensus_peers.cc:487] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:40.300086 447 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 153) took 3559 ms. Trace:
I20250112 02:52:40.300215 447 rpcz_store.cc:276] 0112 02:52:36.740331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.182360 (+3442029us) service_pool.cc:225] Handling call
0112 02:52:40.183191 (+ 831us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:40.183802 (+ 611us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:40.299757 (+115955us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:40.299903 (+ 146us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:40.300018 (+ 115us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:40.185241 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.185314 (+ 73us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.185323 (+ 9us) write_op.cc:435] Acquired schema lock
0112 02:52:40.185326 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:40.185383 (+ 57us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.185390 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.185394 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:40.185477 (+ 83us) tablet.cc:698] Row locks acquired
0112 02:52:40.185479 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:40.185502 (+ 23us) write_op.cc:270] Start()
0112 02:52:40.185522 (+ 20us) write_op.cc:276] Timestamp: P: 1736650360185483 usec, L: 0
0112 02:52:40.185525 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.185634 (+ 109us) log.cc:844] Serialized 433 byte log entry
0112 02:52:40.286898 (+101264us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.298865 (+ 11967us) write_op.cc:301] APPLY: starting
0112 02:52:40.298882 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.298921 (+ 39us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.298955 (+ 34us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.299112 (+ 157us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.299142 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:40.299149 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:40.299227 (+ 78us) log.cc:844] Serialized 29 byte log entry
0112 02:52:40.299630 (+ 403us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.299653 (+ 23us) write_op.cc:454] Released schema lock
0112 02:52:40.299661 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":11876,"num_ops":2,"prepare.queue_time_us":1081,"prepare.run_cpu_time_us":433,"prepare.run_wall_time_us":432,"replication_time_us":101346}]]}
I20250112 02:52:40.302309 1102 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:40.303824 2598 raft_consensus.cc:491] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.303962 2598 raft_consensus.cc:513] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.304617 2598 leader_election.cc:290] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:40.305260 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6f080e5fd84084b601b646d4e6c9e2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:40.305753 855 leader_election.cc:343] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:40.306329 854 leader_election.cc:343] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: ca6f080e5fd84084b601b646d4e6c9e2
I20250112 02:52:40.306420 854 leader_election.cc:304] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:40.306572 2562 ts_tablet_manager.cc:1918] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:40.306679 2562 log.cc:1198] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/7a34d03a80ae402db6607261a5c40b60
I20250112 02:52:40.306725 2598 raft_consensus.cc:2743] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:40.307008 2562 ts_tablet_manager.cc:1939] T 7a34d03a80ae402db6607261a5c40b60 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:40.308280 2706 tablet.cc:1721] T 5cc414d6420c41839621ab858067548b P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:40.309310 1173 tablet_bootstrap.cc:492] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:40.309475 1173 ts_tablet_manager.cc:1397] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.002s
I20250112 02:52:40.310238 1173 raft_consensus.cc:357] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:40.310428 1173 raft_consensus.cc:383] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.310498 1173 raft_consensus.cc:738] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.310774 1173 consensus_queue.cc:260] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:40.311226 1173 ts_tablet_manager.cc:1428] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:40.311452 1173 tablet_bootstrap.cc:492] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:40.309386 1177 ts_tablet_manager.cc:1428] T 3e67f0a0c065498f9a644220414415b0 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20250112 02:52:40.314776 1173 tablet_bootstrap.cc:654] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.315510 1181 tablet_bootstrap.cc:492] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:40.315833 1177 tablet_bootstrap.cc:492] T 42444e99aba84a7b873bc8cdf954a6dc P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:40.325381 1181 tablet_bootstrap.cc:654] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.326589 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 187) took 1493 ms. Trace:
I20250112 02:52:40.326642 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_7f1cd8fa7b844d0fba0cad8d1a6652e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:40.326694 2562 rpcz_store.cc:276] 0112 02:52:38.833569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.833877 (+ 308us) service_pool.cc:225] Handling call
0112 02:52:40.211337 (+1377460us) ts_tablet_manager.cc:1145] Deleting tablet 7a34d03a80ae402db6607261a5c40b60
0112 02:52:40.211345 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.252986 (+ 41641us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.286252 (+ 33266us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.326572 (+ 40320us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1377337}
W20250112 02:52:40.327239 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f1cd8fa7b844d0fba0cad8d1a6652e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:40.328429 2562 tablet_replica.cc:331] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:40.328421 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 7a34d03a80ae402db6607261a5c40b60 (table default.Table_af61980bca5e4995a77da652dcfb3f0d [id=9d7284a74b4148c0a8d20eb220ef9e17]) successfully deleted
I20250112 02:52:40.328608 2562 raft_consensus.cc:2238] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:40.328608 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_0063877a6c23462bb1550693378db102"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:40.328734 2562 raft_consensus.cc:2267] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:40.329063 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0063877a6c23462bb1550693378db102 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:40.329182 2562 ts_tablet_manager.cc:1905] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.330117 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_38abdcafa8f94c1a82760c1c48f93768"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:40.330504 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38abdcafa8f94c1a82760c1c48f93768 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:40.332535 991 consensus_peers.cc:487] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9af55dab202444358ea1a0674d37946d. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:40.333878 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_e7153c9aa3634845b04fa8458133877c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:40.334275 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7153c9aa3634845b04fa8458133877c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:40.335503 965 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:40.336971 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0
I20250112 02:52:40.337392 449 catalog_manager.cc:6596] ReplaceTablet: tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 deleted and replaced by tablet 3dab0466476c4a9aa265271c1abd91a4
I20250112 02:52:40.337507 449 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:40610 (request call id 148) took 4143 ms. Trace:
I20250112 02:52:40.337579 449 rpcz_store.cc:276] 0112 02:52:36.193996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.154748 (+3960752us) service_pool.cc:225] Handling call
0112 02:52:40.337458 (+182710us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:40.196495 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.196599 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.196606 (+ 7us) write_op.cc:435] Acquired schema lock
0112 02:52:40.196610 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:40.196711 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.196722 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.196728 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:40.196804 (+ 76us) tablet.cc:698] Row locks acquired
0112 02:52:40.196808 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:40.196839 (+ 31us) write_op.cc:270] Start()
0112 02:52:40.196867 (+ 28us) write_op.cc:276] Timestamp: P: 1736650360196815 usec, L: 0
0112 02:52:40.196872 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.197054 (+ 182us) log.cc:844] Serialized 622 byte log entry
0112 02:52:40.336271 (+139217us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.336394 (+ 123us) write_op.cc:301] APPLY: starting
0112 02:52:40.336406 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.336443 (+ 37us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.336478 (+ 35us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.336550 (+ 72us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.336590 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:40.336596 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:40.336644 (+ 48us) log.cc:844] Serialized 29 byte log entry
0112 02:52:40.336689 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.336701 (+ 12us) write_op.cc:454] Released schema lock
0112 02:52:40.336706 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:40.336772 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.336775 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":75,"apply.run_cpu_time_us":365,"apply.run_wall_time_us":365,"num_ops":2,"prepare.queue_time_us":4127,"prepare.run_cpu_time_us":643,"prepare.run_wall_time_us":643,"replication_time_us":139376}]]}
I20250112 02:52:40.338258 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_8bf5e22d7ca64a46b04eecfa87d1a646"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:40.338615 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8bf5e22d7ca64a46b04eecfa87d1a646 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:40.339910 450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 154) took 3599 ms. Trace:
I20250112 02:52:40.339989 450 rpcz_store.cc:276] 0112 02:52:36.740425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.192996 (+3452571us) service_pool.cc:225] Handling call
0112 02:52:40.193928 (+ 932us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:40.194434 (+ 506us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:40.339660 (+145226us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:40.339770 (+ 110us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:40.339857 (+ 87us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:40.197157 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.197254 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.197260 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:40.197264 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:40.197345 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.197354 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.197360 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:40.197430 (+ 70us) tablet.cc:698] Row locks acquired
0112 02:52:40.197434 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:40.197459 (+ 25us) write_op.cc:270] Start()
0112 02:52:40.197484 (+ 25us) write_op.cc:276] Timestamp: P: 1736650360197437 usec, L: 0
0112 02:52:40.197489 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.197621 (+ 132us) log.cc:844] Serialized 433 byte log entry
0112 02:52:40.336324 (+138703us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.336832 (+ 508us) write_op.cc:301] APPLY: starting
0112 02:52:40.336841 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.336866 (+ 25us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.336888 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.336952 (+ 64us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.336984 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:40.336990 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:40.337030 (+ 40us) log.cc:844] Serialized 29 byte log entry
0112 02:52:40.337060 (+ 30us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.337070 (+ 10us) write_op.cc:454] Released schema lock
0112 02:52:40.337074 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":480,"apply.run_cpu_time_us":288,"apply.run_wall_time_us":288,"num_ops":2,"prepare.queue_time_us":2124,"prepare.run_cpu_time_us":513,"prepare.run_wall_time_us":512,"replication_time_us":138821}]]}
I20250112 02:52:40.340587 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_22969d80f802435898d7d60983f0daf9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:40.340945 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22969d80f802435898d7d60983f0daf9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:40.341567 2003 ts_tablet_manager.cc:1918] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:40.342144 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 with delete_type TABLET_DATA_DELETED (replaced by tablet 3dab0466476c4a9aa265271c1abd91a4) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:40.342850 2003 log.cc:1198] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/f31b777e7e0f43a28839d8c48c6a6590
I20250112 02:52:40.342900 826 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:40.346385 2003 ts_tablet_manager.cc:1939] T f31b777e7e0f43a28839d8c48c6a6590 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:40.343042 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 with delete_type TABLET_DATA_DELETED (replaced by tablet 3dab0466476c4a9aa265271c1abd91a4) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:40.348026 2024 ts_tablet_manager.cc:1918] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:40.348140 2024 log.cc:1198] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/4b599e983ec04f53a440d1ab828d12ab
I20250112 02:52:40.348709 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 with delete_type TABLET_DATA_DELETED (replaced by tablet 3dab0466476c4a9aa265271c1abd91a4) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:40.349846 1177 tablet_bootstrap.cc:654] T 42444e99aba84a7b873bc8cdf954a6dc P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:40.350569 991 consensus_peers.cc:487] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe614eaeb793473d9b2da206b7345562. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:52:40.352663 989 consensus_peers.cc:487] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:40.356612 2024 ts_tablet_manager.cc:1939] T 4b599e983ec04f53a440d1ab828d12ab P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:40.356916 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 86) took 5822 ms. Trace:
I20250112 02:52:40.357024 2024 rpcz_store.cc:276] 0112 02:52:34.534581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.534646 (+ 65us) service_pool.cc:225] Handling call
0112 02:52:40.255573 (+5720927us) ts_tablet_manager.cc:1145] Deleting tablet 4b599e983ec04f53a440d1ab828d12ab
0112 02:52:40.255581 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.277461 (+ 21880us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.347936 (+ 70475us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.356902 (+ 8966us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5712520}
I20250112 02:52:40.357450 2024 tablet_replica.cc:331] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:40.357633 2024 raft_consensus.cc:2238] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:40.357790 2024 raft_consensus.cc:2267] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:40.358273 2024 ts_tablet_manager.cc:1905] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.358791 2003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 259) took 1008 ms. Trace:
I20250112 02:52:40.358937 2003 rpcz_store.cc:276] 0112 02:52:39.343454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.343532 (+ 78us) service_pool.cc:225] Handling call
0112 02:52:40.187785 (+844253us) ts_tablet_manager.cc:1145] Deleting tablet f31b777e7e0f43a28839d8c48c6a6590
0112 02:52:40.187795 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.206787 (+ 18992us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.340927 (+134140us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.352229 (+ 11302us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":844077}
I20250112 02:52:40.359722 2003 tablet_replica.cc:331] T f924459d1150472aafb325a11825db23 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:40.359959 2003 raft_consensus.cc:2238] T f924459d1150472aafb325a11825db23 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:40.360697 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet f31b777e7e0f43a28839d8c48c6a6590 (table default.Table_41b7056b6e9240928d0af52aec2e9338 [id=6402abce8d1d45c182cc989490c6a5f0]) successfully deleted
I20250112 02:52:40.361023 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 4b599e983ec04f53a440d1ab828d12ab (table default.Table_feacb3254bbe4765a4e0a4908fbc694c [id=f396d2dc8dbd47729f49f7511467d481]) successfully deleted
I20250112 02:52:40.362160 1181 tablet_bootstrap.cc:492] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:40.362280 1181 ts_tablet_manager.cc:1397] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.001s
I20250112 02:52:40.363166 1181 raft_consensus.cc:357] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:40.363332 1181 raft_consensus.cc:383] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.363379 1181 raft_consensus.cc:738] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.363611 1181 consensus_queue.cc:260] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:40.364614 453 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 155) took 3589 ms. Trace:
I20250112 02:52:40.364720 453 rpcz_store.cc:276] 0112 02:52:36.775359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.202093 (+3426734us) service_pool.cc:225] Handling call
0112 02:52:40.326539 (+124446us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd72d03 00007f3e13e75236 00007f3e13e74f65 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:40.327345 (+ 806us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:40.327823 (+ 478us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:40.358120 (+ 30297us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:40.364485 (+ 6365us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:40.364579 (+ 94us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:40.331506 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.331600 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.331606 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:40.331608 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:40.331681 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.331689 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.331693 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:40.331752 (+ 59us) tablet.cc:698] Row locks acquired
0112 02:52:40.331754 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:40.331788 (+ 34us) write_op.cc:270] Start()
0112 02:52:40.331831 (+ 43us) write_op.cc:276] Timestamp: P: 1736650360331739 usec, L: 0
0112 02:52:40.331834 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.332005 (+ 171us) log.cc:844] Serialized 433 byte log entry
0112 02:52:40.356890 (+ 24885us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.357046 (+ 156us) write_op.cc:301] APPLY: starting
0112 02:52:40.357059 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.357109 (+ 50us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.357141 (+ 32us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.357236 (+ 95us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.357279 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:40.357286 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:40.357347 (+ 61us) log.cc:844] Serialized 29 byte log entry
0112 02:52:40.357403 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.357418 (+ 15us) write_op.cc:454] Released schema lock
0112 02:52:40.357424 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:40.357486 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.357492 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":285502208,"child_traces":[["op",{"apply.queue_time_us":109,"apply.run_cpu_time_us":434,"apply.run_wall_time_us":433,"num_ops":2,"prepare.queue_time_us":3162,"prepare.run_cpu_time_us":602,"prepare.run_wall_time_us":602,"raft.queue_time_us":550,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":248,"replication_time_us":25031}]]}
I20250112 02:52:40.365459 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_bb82d19aab034576ad7a0303219fa410"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:40.365926 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb82d19aab034576ad7a0303219fa410 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:40.366930 455 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 156) took 3422 ms. Trace:
I20250112 02:52:40.367035 455 rpcz_store.cc:276] 0112 02:52:36.944314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.240664 (+3296350us) service_pool.cc:225] Handling call
0112 02:52:40.328511 (+ 87847us) spinlock_profiling.cc:243] Waited 87.5 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd72d03 00007f3e13e75236 00007f3e13e74f65 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:40.329158 (+ 647us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:40.329521 (+ 363us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:40.358225 (+ 28704us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:40.366822 (+ 8597us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:40.366901 (+ 79us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:40.332119 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.332203 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.332208 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:40.332211 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:40.332280 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.332288 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.332292 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:40.332345 (+ 53us) tablet.cc:698] Row locks acquired
0112 02:52:40.332348 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:40.332372 (+ 24us) write_op.cc:270] Start()
0112 02:52:40.332395 (+ 23us) write_op.cc:276] Timestamp: P: 1736650360332323 usec, L: 0
0112 02:52:40.332399 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.332537 (+ 138us) log.cc:844] Serialized 433 byte log entry
0112 02:52:40.356959 (+ 24422us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.357548 (+ 589us) write_op.cc:301] APPLY: starting
0112 02:52:40.357557 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.357582 (+ 25us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.357607 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.357685 (+ 78us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.357714 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:40.357719 (+ 5us) write_op.cc:312] APPLY: finished
0112 02:52:40.357763 (+ 44us) log.cc:844] Serialized 29 byte log entry
0112 02:52:40.357822 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.357834 (+ 12us) write_op.cc:454] Released schema lock
0112 02:52:40.357841 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:40.357899 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.357902 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":201331584,"child_traces":[["op",{"apply.queue_time_us":561,"apply.run_cpu_time_us":344,"apply.run_wall_time_us":345,"num_ops":2,"prepare.queue_time_us":2227,"prepare.run_cpu_time_us":492,"prepare.run_wall_time_us":492,"raft.queue_time_us":337,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":24543}]]}
I20250112 02:52:40.367709 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_6671f628f89b428a83eca77e9ca26516" } modify_external_catalogs: true
I20250112 02:52:40.367921 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_6671f628f89b428a83eca77e9ca26516" } modify_external_catalogs: true
I20250112 02:52:40.370085 2703 tablet.cc:1721] T 5cc414d6420c41839621ab858067548b P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:40.374964 454 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 157) took 3430 ms. Trace:
I20250112 02:52:40.375075 454 rpcz_store.cc:276] 0112 02:52:36.944510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.270362 (+3325852us) service_pool.cc:225] Handling call
0112 02:52:40.330047 (+ 59685us) spinlock_profiling.cc:243] Waited 59.4 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd72d03 00007f3e13e75236 00007f3e13e74f65 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:40.330574 (+ 527us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:40.330927 (+ 353us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:40.374757 (+ 43830us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:40.374860 (+ 103us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:40.374934 (+ 74us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:40.332641 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.332710 (+ 69us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.332715 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:40.332717 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:40.332807 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.332814 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.332819 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:40.332874 (+ 55us) tablet.cc:698] Row locks acquired
0112 02:52:40.332876 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:40.332901 (+ 25us) write_op.cc:270] Start()
0112 02:52:40.332962 (+ 61us) write_op.cc:276] Timestamp: P: 1736650360332852 usec, L: 0
0112 02:52:40.332974 (+ 12us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.333106 (+ 132us) log.cc:844] Serialized 433 byte log entry
0112 02:52:40.373858 (+ 40752us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.373984 (+ 126us) write_op.cc:301] APPLY: starting
0112 02:52:40.373995 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.374029 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.374051 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.374135 (+ 84us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.374162 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:40.374169 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:40.374229 (+ 60us) log.cc:844] Serialized 29 byte log entry
0112 02:52:40.374275 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.374291 (+ 16us) write_op.cc:454] Released schema lock
0112 02:52:40.374297 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:40.374350 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.374354 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":136690944,"child_traces":[["op",{"apply.queue_time_us":77,"apply.run_cpu_time_us":367,"apply.run_wall_time_us":367,"num_ops":2,"prepare.queue_time_us":1396,"prepare.run_cpu_time_us":500,"prepare.run_wall_time_us":499,"replication_time_us":40861}]]}
I20250112 02:52:40.376061 447 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 158) took 3431 ms. Trace:
I20250112 02:52:40.376152 447 rpcz_store.cc:276] 0112 02:52:36.944825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.300817 (+3355992us) service_pool.cc:225] Handling call
0112 02:52:40.333794 (+ 32977us) spinlock_profiling.cc:243] Waited 32.7 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd72d03 00007f3e13e75236 00007f3e13e74f65 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:40.334373 (+ 579us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:40.334697 (+ 324us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:40.375847 (+ 41150us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:40.375941 (+ 94us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:40.376011 (+ 70us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:40.335489 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.335559 (+ 70us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.335564 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:40.335567 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:40.335643 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.335650 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.335657 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:40.335709 (+ 52us) tablet.cc:698] Row locks acquired
0112 02:52:40.335712 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:40.335737 (+ 25us) write_op.cc:270] Start()
0112 02:52:40.335757 (+ 20us) write_op.cc:276] Timestamp: P: 1736650360335687 usec, L: 0
0112 02:52:40.335760 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.335875 (+ 115us) log.cc:844] Serialized 433 byte log entry
0112 02:52:40.373918 (+ 38043us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.374408 (+ 490us) write_op.cc:301] APPLY: starting
0112 02:52:40.374419 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.374445 (+ 26us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.374468 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.374538 (+ 70us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.374563 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:40.374569 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:40.374612 (+ 43us) log.cc:844] Serialized 29 byte log entry
0112 02:52:40.374645 (+ 33us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.374656 (+ 11us) write_op.cc:454] Released schema lock
0112 02:52:40.374661 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":75301632,"child_traces":[["op",{"apply.queue_time_us":460,"apply.run_cpu_time_us":301,"apply.run_wall_time_us":301,"num_ops":2,"prepare.queue_time_us":434,"prepare.run_cpu_time_us":462,"prepare.run_wall_time_us":462,"replication_time_us":38143}]]}
I20250112 02:52:40.378141 449 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 159) took 3402 ms. Trace:
I20250112 02:52:40.378245 449 rpcz_store.cc:276] 0112 02:52:36.975241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.338088 (+3362847us) service_pool.cc:225] Handling call
0112 02:52:40.338698 (+ 610us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:40.339056 (+ 358us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:40.377953 (+ 38897us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:40.378037 (+ 84us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:40.378093 (+ 56us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:40.348642 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.348725 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.348732 (+ 7us) write_op.cc:435] Acquired schema lock
0112 02:52:40.348734 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:40.348812 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.348821 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.348828 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:40.348889 (+ 61us) tablet.cc:698] Row locks acquired
0112 02:52:40.348891 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:40.348914 (+ 23us) write_op.cc:270] Start()
0112 02:52:40.348936 (+ 22us) write_op.cc:276] Timestamp: P: 1736650360348863 usec, L: 0
0112 02:52:40.348938 (+ 2us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.349066 (+ 128us) log.cc:844] Serialized 433 byte log entry
0112 02:52:40.377415 (+ 28349us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.377478 (+ 63us) write_op.cc:301] APPLY: starting
0112 02:52:40.377489 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.377515 (+ 26us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.377539 (+ 24us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.377613 (+ 74us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.377636 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:40.377641 (+ 5us) write_op.cc:312] APPLY: finished
0112 02:52:40.377689 (+ 48us) log.cc:844] Serialized 29 byte log entry
0112 02:52:40.377731 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.377741 (+ 10us) write_op.cc:454] Released schema lock
0112 02:52:40.377746 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":30,"apply.run_cpu_time_us":309,"apply.run_wall_time_us":309,"num_ops":2,"prepare.queue_time_us":9230,"prepare.run_cpu_time_us":486,"prepare.run_wall_time_us":487,"replication_time_us":28457}]]}
I20250112 02:52:40.380252 450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 160) took 3404 ms. Trace:
I20250112 02:52:40.380394 450 rpcz_store.cc:276] 0112 02:52:36.975336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.340422 (+3365086us) service_pool.cc:225] Handling call
0112 02:52:40.341013 (+ 591us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:40.341342 (+ 329us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:40.380025 (+ 38683us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:40.380121 (+ 96us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:40.380195 (+ 74us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:40.349202 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.349278 (+ 76us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.349283 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:40.349285 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:40.349342 (+ 57us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.349349 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.349353 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:40.349401 (+ 48us) tablet.cc:698] Row locks acquired
0112 02:52:40.349403 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:40.349423 (+ 20us) write_op.cc:270] Start()
0112 02:52:40.349443 (+ 20us) write_op.cc:276] Timestamp: P: 1736650360349373 usec, L: 0
0112 02:52:40.349446 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.349546 (+ 100us) log.cc:844] Serialized 433 byte log entry
0112 02:52:40.379574 (+ 30028us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.379642 (+ 68us) write_op.cc:301] APPLY: starting
0112 02:52:40.379652 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.379680 (+ 28us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.379699 (+ 19us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.379769 (+ 70us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.379792 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:40.379800 (+ 8us) write_op.cc:312] APPLY: finished
0112 02:52:40.379847 (+ 47us) log.cc:844] Serialized 29 byte log entry
0112 02:52:40.379887 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.379899 (+ 12us) write_op.cc:454] Released schema lock
0112 02:52:40.379904 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":31,"apply.run_cpu_time_us":308,"apply.run_wall_time_us":309,"num_ops":2,"prepare.queue_time_us":7572,"prepare.run_cpu_time_us":396,"prepare.run_wall_time_us":396,"replication_time_us":30103}]]}
I20250112 02:52:40.380991 450 master_service.cc:946] ReplaceTablet: received request to replace tablet daef3a58605e40978f7389698b22265d from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:40.382198 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_cb4ed576320c497588e65dcbac70afe9" } modify_external_catalogs: true
I20250112 02:52:40.382434 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_cb4ed576320c497588e65dcbac70afe9" } modify_external_catalogs: true
I20250112 02:52:40.385882 2710 tablet.cc:1721] T 5cc414d6420c41839621ab858067548b P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:40.387673 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_dd77b8845fdd46b7a2debbadaf553f75" } new_table_name: "default.Table_2fcdbb05970f4a098f7ba3e287fa0348" modify_external_catalogs: true
I20250112 02:52:40.389413 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_6b8cb70a5e5b4519af5d247f6b5bb8b0" } modify_external_catalogs: true
I20250112 02:52:40.389556 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_6b8cb70a5e5b4519af5d247f6b5bb8b0" } modify_external_catalogs: true
I20250112 02:52:40.391496 2712 tablet.cc:1721] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:40.396327 1173 tablet_bootstrap.cc:492] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:40.396485 1173 ts_tablet_manager.cc:1397] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.085s user 0.000s sys 0.004s
I20250112 02:52:40.397217 1173 raft_consensus.cc:357] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.403661 1181 ts_tablet_manager.cc:1428] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.041s user 0.001s sys 0.001s
I20250112 02:52:40.406242 1173 raft_consensus.cc:383] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.406877 1173 raft_consensus.cc:738] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.407750 1173 consensus_queue.cc:260] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.413599 1181 tablet_bootstrap.cc:492] T 13578542e9e741e8817e0ee63b10f1a1 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:40.415354 1181 tablet_bootstrap.cc:654] T 13578542e9e741e8817e0ee63b10f1a1 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.416661 1173 ts_tablet_manager.cc:1428] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250112 02:52:40.402967 2228 raft_consensus.cc:491] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.423923 2228 raft_consensus.cc:513] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.424559 2228 leader_election.cc:290] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.425105 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9c845571f44c8eb8753090d63df45f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:40.425658 991 leader_election.cc:343] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:40.421607 2326 consensus_queue.cc:1035] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:40.426246 1173 tablet_bootstrap.cc:492] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:40.428896 1173 tablet_bootstrap.cc:654] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.434276 2024 ts_tablet_manager.cc:1918] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 02:52:40.435626 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet ca2165362ee541adac724b63f39b1489 (table default.Table_41b7056b6e9240928d0af52aec2e9338 [id=6402abce8d1d45c182cc989490c6a5f0], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet ca2165362ee541adac724b63f39b1489 consensus state has no leader
I20250112 02:52:40.423782 966 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:40.445255 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9c845571f44c8eb8753090d63df45f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:40.445430 917 raft_consensus.cc:2463] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:40.406111 2562 ts_tablet_manager.cc:1918] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:40.447667 2562 log.cc:1198] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/9af55dab202444358ea1a0674d37946d
I20250112 02:52:40.448060 2562 ts_tablet_manager.cc:1939] T 9af55dab202444358ea1a0674d37946d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:40.448402 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 188) took 1614 ms. Trace:
I20250112 02:52:40.448555 2562 rpcz_store.cc:276] 0112 02:52:38.834138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.834282 (+ 144us) service_pool.cc:225] Handling call
0112 02:52:40.328383 (+1494101us) ts_tablet_manager.cc:1145] Deleting tablet 9af55dab202444358ea1a0674d37946d
0112 02:52:40.328395 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.331937 (+ 3542us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.406023 (+ 74086us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.448386 (+ 42363us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1493985}
I20250112 02:52:40.449177 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 189) took 1614 ms. Trace:
I20250112 02:52:40.449299 2562 rpcz_store.cc:276] 0112 02:52:38.834249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.834878 (+ 629us) service_pool.cc:225] Handling call
0112 02:52:40.449039 (+1614161us) ts_tablet_manager.cc:1145] Deleting tablet 058a9ca089cd4967887b0a1127c7d03f
0112 02:52:40.449055 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.449141 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1614040}
I20250112 02:52:40.449720 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 190) took 1614 ms. Trace:
I20250112 02:52:40.449674 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 9af55dab202444358ea1a0674d37946d (table default.Table_26dfe346379948b39f90b31653c2a463 [id=6c6f30e60e79451397841d090d11d8b7]) successfully deleted
I20250112 02:52:40.449846 2562 rpcz_store.cc:276] 0112 02:52:38.835095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.835323 (+ 228us) service_pool.cc:225] Handling call
0112 02:52:40.449634 (+1614311us) ts_tablet_manager.cc:1145] Deleting tablet d45fce86e02f46d4a57af718fa2a62c8
0112 02:52:40.449642 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.449688 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1614196}
W20250112 02:52:40.450047 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 058a9ca089cd4967887b0a1127c7d03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 058a9ca089cd4967887b0a1127c7d03f already in progress: creating tablet
I20250112 02:52:40.450194 2003 raft_consensus.cc:2267] T f924459d1150472aafb325a11825db23 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:40.450237 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 191) took 1614 ms. Trace:
I20250112 02:52:40.450388 2562 rpcz_store.cc:276] 0112 02:52:38.835570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.835864 (+ 294us) service_pool.cc:225] Handling call
0112 02:52:40.450164 (+1614300us) ts_tablet_manager.cc:1145] Deleting tablet 7a18132549374218893f35a4e4226b72
0112 02:52:40.450172 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.450213 (+ 41us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1614128}
I20250112 02:52:40.450764 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 192) took 1614 ms. Trace:
I20250112 02:52:40.450862 2003 ts_tablet_manager.cc:1905] T f924459d1150472aafb325a11825db23 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.450881 2562 rpcz_store.cc:276] 0112 02:52:38.836109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.836340 (+ 231us) service_pool.cc:225] Handling call
0112 02:52:40.450691 (+1614351us) ts_tablet_manager.cc:1145] Deleting tablet 003a3b4705e943c6a900762ec11ddd47
0112 02:52:40.450698 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.450740 (+ 42us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1614216}
I20250112 02:52:40.451277 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 193) took 1614 ms. Trace:
W20250112 02:52:40.455364 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet d45fce86e02f46d4a57af718fa2a62c8 because the tablet was not found. No further retry: Not found: Tablet not found: d45fce86e02f46d4a57af718fa2a62c8
W20250112 02:52:40.455691 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 7a18132549374218893f35a4e4226b72 because the tablet was not found. No further retry: Not found: Tablet not found: 7a18132549374218893f35a4e4226b72
W20250112 02:52:40.455906 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 003a3b4705e943c6a900762ec11ddd47 because the tablet was not found. No further retry: Not found: Tablet not found: 003a3b4705e943c6a900762ec11ddd47
I20250112 02:52:40.457664 1181 tablet_bootstrap.cc:492] T 13578542e9e741e8817e0ee63b10f1a1 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:40.457918 1181 ts_tablet_manager.cc:1397] T 13578542e9e741e8817e0ee63b10f1a1 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20250112 02:52:40.458581 453 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 161) took 3434 ms. Trace:
I20250112 02:52:40.458575 1181 raft_consensus.cc:357] T 13578542e9e741e8817e0ee63b10f1a1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:40.458762 1181 raft_consensus.cc:383] T 13578542e9e741e8817e0ee63b10f1a1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.458822 1181 raft_consensus.cc:738] T 13578542e9e741e8817e0ee63b10f1a1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.458676 453 rpcz_store.cc:276] 0112 02:52:37.023855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.365274 (+3341419us) service_pool.cc:225] Handling call
0112 02:52:40.366002 (+ 728us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:40.366384 (+ 382us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:40.458366 (+ 91982us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:40.458471 (+ 105us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:40.458549 (+ 78us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:40.368915 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.369002 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.369006 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:40.369008 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:40.369081 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.369090 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.369096 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:40.369160 (+ 64us) tablet.cc:698] Row locks acquired
0112 02:52:40.369164 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:40.369190 (+ 26us) write_op.cc:270] Start()
0112 02:52:40.369215 (+ 25us) write_op.cc:276] Timestamp: P: 1736650360369133 usec, L: 0
0112 02:52:40.369219 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.369345 (+ 126us) log.cc:844] Serialized 433 byte log entry
0112 02:52:40.456355 (+ 87010us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.456520 (+ 165us) write_op.cc:301] APPLY: starting
0112 02:52:40.456538 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.456580 (+ 42us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.456602 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.456737 (+ 135us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.456770 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:40.456777 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:40.456857 (+ 80us) log.cc:844] Serialized 29 byte log entry
0112 02:52:40.456923 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.456942 (+ 19us) write_op.cc:454] Released schema lock
0112 02:52:40.456950 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:40.457041 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.457062 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":114,"apply.run_cpu_time_us":515,"apply.run_wall_time_us":515,"num_ops":2,"prepare.queue_time_us":2150,"prepare.run_cpu_time_us":487,"prepare.run_wall_time_us":487,"replication_time_us":87106}]]}
I20250112 02:52:40.459017 1181 consensus_queue.cc:260] T 13578542e9e741e8817e0ee63b10f1a1 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:40.459501 1181 ts_tablet_manager.cc:1428] T 13578542e9e741e8817e0ee63b10f1a1 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:40.459760 1181 tablet_bootstrap.cc:492] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:40.451396 2562 rpcz_store.cc:276] 0112 02:52:38.836611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.836911 (+ 300us) service_pool.cc:225] Handling call
0112 02:52:40.451202 (+1614291us) ts_tablet_manager.cc:1145] Deleting tablet 2730634a5cfd4c2fb5c4e30100c556d4
0112 02:52:40.451210 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.451252 (+ 42us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1614160}
I20250112 02:52:40.445750 989 leader_election.cc:304] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:40.401216 2710 tablet.cc:1721] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 02:52:40.456604 991 consensus_peers.cc:487] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:40.471652 2228 raft_consensus.cc:2798] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:40.471731 2228 raft_consensus.cc:491] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:40.471791 2228 raft_consensus.cc:3054] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:40.473613 2451 raft_consensus.cc:491] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.473762 2451 raft_consensus.cc:513] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.474195 2228 raft_consensus.cc:513] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.474279 2451 leader_election.cc:290] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.474642 2228 leader_election.cc:290] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.475435 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9c845571f44c8eb8753090d63df45f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
W20250112 02:52:40.476289 991 leader_election.cc:343] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:40.477381 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9c845571f44c8eb8753090d63df45f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:40.477535 918 raft_consensus.cc:3054] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:40.479905 918 raft_consensus.cc:2463] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
W20250112 02:52:40.480151 713 leader_election.cc:343] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 06edfca60eed4dd3a2dee7b662b0970c
I20250112 02:52:40.486264 2003 ts_tablet_manager.cc:1918] T f924459d1150472aafb325a11825db23 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:40.486312 989 leader_election.cc:304] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:40.487298 2024 log.cc:1198] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/352c87f1dc724e7b94bc2e64abfce8d7
I20250112 02:52:40.487632 2024 ts_tablet_manager.cc:1939] T 352c87f1dc724e7b94bc2e64abfce8d7 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:40.486822 452 catalog_manager.cc:5526] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
W20250112 02:52:40.489956 2381 log.cc:926] Time spent T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Append to log took a long time: real 0.080s user 0.000s sys 0.003s
I20250112 02:52:40.490391 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:40.491091 1181 tablet_bootstrap.cc:654] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.491732 2451 consensus_queue.cc:1035] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:40.491648 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:40.496783 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:40.486912 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06edfca60eed4dd3a2dee7b662b0970c" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:40.498323 1050 raft_consensus.cc:2463] T 06edfca60eed4dd3a2dee7b662b0970c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:40.498818 1101 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:40.499023 714 leader_election.cc:304] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:40.499359 2451 raft_consensus.cc:2798] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:40.499426 2451 raft_consensus.cc:491] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:40.499522 2451 raft_consensus.cc:3054] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:40.498490 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:40.501967 2451 raft_consensus.cc:513] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.502549 2451 leader_election.cc:290] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.503094 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06edfca60eed4dd3a2dee7b662b0970c" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:40.503249 1050 raft_consensus.cc:3054] T 06edfca60eed4dd3a2dee7b662b0970c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:40.504390 713 leader_election.cc:343] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 06edfca60eed4dd3a2dee7b662b0970c
W20250112 02:52:40.506227 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 51432us
I20250112 02:52:40.507290 1181 tablet_bootstrap.cc:492] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:40.508958 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:40.509517 2445 consensus_queue.cc:1035] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:40.513692 1177 tablet_bootstrap.cc:492] T 42444e99aba84a7b873bc8cdf954a6dc P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:40.513839 1177 ts_tablet_manager.cc:1397] T 42444e99aba84a7b873bc8cdf954a6dc P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.198s user 0.004s sys 0.006s
I20250112 02:52:40.514499 1177 raft_consensus.cc:357] T 42444e99aba84a7b873bc8cdf954a6dc P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.514663 1177 raft_consensus.cc:383] T 42444e99aba84a7b873bc8cdf954a6dc P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.514721 1177 raft_consensus.cc:738] T 42444e99aba84a7b873bc8cdf954a6dc P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.514923 1177 consensus_queue.cc:260] T 42444e99aba84a7b873bc8cdf954a6dc P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.515408 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:40.517457 1177 ts_tablet_manager.cc:1428] T 42444e99aba84a7b873bc8cdf954a6dc P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 02:52:40.517712 1177 tablet_bootstrap.cc:492] T 46d1beac4410422bbf1e912d35999f3d P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:40.513057 2696 raft_consensus.cc:491] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.518729 2696 raft_consensus.cc:513] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:40.519022 2228 raft_consensus.cc:2798] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:40.519121 1177 tablet_bootstrap.cc:654] T 46d1beac4410422bbf1e912d35999f3d P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.519140 2228 raft_consensus.cc:695] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:40.519359 2696 leader_election.cc:290] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:40.519346 2228 consensus_queue.cc:237] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.520205 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3a4e76b724452c8b238aae0198e297" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:40.520442 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
W20250112 02:52:40.520906 855 leader_election.cc:343] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:40.521169 1181 ts_tablet_manager.cc:1397] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.061s user 0.001s sys 0.004s
I20250112 02:52:40.522259 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3a4e76b724452c8b238aae0198e297" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:40.521695 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
W20250112 02:52:40.524029 854 leader_election.cc:343] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:40.524138 854 leader_election.cc:304] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:40.524477 2696 raft_consensus.cc:2743] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:40.521960 1181 raft_consensus.cc:357] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:40.525961 1181 raft_consensus.cc:383] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.526146 1181 raft_consensus.cc:738] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.527568 2451 consensus_queue.cc:1035] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:40.531932 2609 consensus_queue.cc:1035] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.049s
I20250112 02:52:40.532536 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 1 got 0
I20250112 02:52:40.537927 452 catalog_manager.cc:5526] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:40.527230 1181 consensus_queue.cc:260] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:40.542508 2047 consensus_queue.cc:1035] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:40.543094 1181 ts_tablet_manager.cc:1428] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20250112 02:52:40.543617 2609 consensus_queue.cc:1035] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:40.543522 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:40.544490 1181 tablet_bootstrap.cc:492] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:40.556134 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:40.560388 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:40.561920 1181 tablet_bootstrap.cc:654] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.564927 2696 raft_consensus.cc:491] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.566915 2696 raft_consensus.cc:513] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.568156 2696 leader_election.cc:290] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:40.567199 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
W20250112 02:52:40.570586 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 2730634a5cfd4c2fb5c4e30100c556d4 because the tablet was not found. No further retry: Not found: Tablet not found: 2730634a5cfd4c2fb5c4e30100c556d4
I20250112 02:52:40.571367 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 194) took 1734 ms. Trace:
I20250112 02:52:40.571378 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5079f87652a4ea58b366d1f9448fd4e" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:40.571748 2562 rpcz_store.cc:276] 0112 02:52:38.837168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.837418 (+ 250us) service_pool.cc:225] Handling call
0112 02:52:40.571278 (+1733860us) ts_tablet_manager.cc:1145] Deleting tablet b61f16f3d601436bb4b32344de28c1a7
0112 02:52:40.571287 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.571353 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1733732}
W20250112 02:52:40.571940 855 leader_election.cc:343] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:40.572196 2562 tablet_replica.cc:331] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
W20250112 02:52:40.572250 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet b61f16f3d601436bb4b32344de28c1a7 because the tablet was not found. No further retry: Not found: Tablet not found: b61f16f3d601436bb4b32344de28c1a7
I20250112 02:52:40.572477 2562 raft_consensus.cc:2238] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:40.572662 2562 raft_consensus.cc:2267] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:40.572903 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:40.573331 2562 ts_tablet_manager.cc:1905] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.574821 452 catalog_manager.cc:5526] T e2f1f1a76e5f4308bf56dd647f02c6d8 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:40.579985 2326 raft_consensus.cc:491] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.580107 2326 raft_consensus.cc:513] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.580652 2326 leader_election.cc:290] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.581115 2562 ts_tablet_manager.cc:1918] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:40.581192 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c821ea3ca4946c290c17135366e0cac" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:40.581650 991 leader_election.cc:343] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:40.582016 1173 tablet_bootstrap.cc:492] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:40.582194 1173 ts_tablet_manager.cc:1397] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.156s user 0.000s sys 0.016s
I20250112 02:52:40.582711 2326 raft_consensus.cc:491] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.582772 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5079f87652a4ea58b366d1f9448fd4e" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:40.582839 2326 raft_consensus.cc:513] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.582612 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:40.583376 2326 leader_election.cc:290] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.583935 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a437952889494b0e820e41fdfffe708d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:40.584092 774 raft_consensus.cc:2463] T a437952889494b0e820e41fdfffe708d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:40.584471 991 leader_election.cc:304] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:40.584385 1173 raft_consensus.cc:357] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.584621 1173 raft_consensus.cc:383] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.584681 1173 raft_consensus.cc:738] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.584769 2208 raft_consensus.cc:2798] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:40.584839 2208 raft_consensus.cc:491] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:40.584897 2208 raft_consensus.cc:3054] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:40.584954 1173 consensus_queue.cc:260] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.585969 1173 ts_tablet_manager.cc:1428] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 02:52:40.586221 1173 tablet_bootstrap.cc:492] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:40.586170 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:40.587549 2208 raft_consensus.cc:513] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.588166 2208 leader_election.cc:290] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.588708 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a437952889494b0e820e41fdfffe708d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:40.588893 774 raft_consensus.cc:3054] T a437952889494b0e820e41fdfffe708d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:40.588582 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:40.589351 1173 tablet_bootstrap.cc:654] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.590111 2562 log.cc:1198] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/f776e2a6ec704cf9b64483bc7d5b1dff
I20250112 02:52:40.590413 2562 ts_tablet_manager.cc:1939] T f776e2a6ec704cf9b64483bc7d5b1dff P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:40.581506 1181 tablet_bootstrap.cc:492] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:40.588572 2696 raft_consensus.cc:491] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.590689 2696 raft_consensus.cc:513] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:40.591264 2696 leader_election.cc:290] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:40.591701 1181 ts_tablet_manager.cc:1397] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.003s
I20250112 02:52:40.591167 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 5cc414d6420c41839621ab858067548b (table default.Table_8fe6c818f15341e08fe8870664eac4c1 [id=268dae304c9143a881c7a0898632a3e4]). Expected version 2 got 1
I20250112 02:52:40.592028 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058a9ca089cd4967887b0a1127c7d03f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:40.592506 855 leader_election.cc:343] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:40.593433 774 raft_consensus.cc:2463] T a437952889494b0e820e41fdfffe708d P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:40.593837 991 leader_election.cc:304] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:40.593477 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:40.594102 2609 raft_consensus.cc:2798] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:40.594201 2609 raft_consensus.cc:695] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:40.594442 2609 consensus_queue.cc:237] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:40.599323 854 leader_election.cc:343] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:40.599419 854 leader_election.cc:304] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:40.598804 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 195) took 1717 ms. Trace:
I20250112 02:52:40.599782 2696 raft_consensus.cc:2743] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:40.600390 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a437952889494b0e820e41fdfffe708d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:40.600562 917 raft_consensus.cc:2463] T a437952889494b0e820e41fdfffe708d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:40.599812 2562 rpcz_store.cc:276] 0112 02:52:38.880944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.880989 (+ 45us) service_pool.cc:225] Handling call
0112 02:52:40.572111 (+1691122us) ts_tablet_manager.cc:1145] Deleting tablet f776e2a6ec704cf9b64483bc7d5b1dff
0112 02:52:40.572122 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.578051 (+ 5929us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.581034 (+ 2983us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.598781 (+ 17747us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1690962}
I20250112 02:52:40.601751 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet f776e2a6ec704cf9b64483bc7d5b1dff (table default.Table_d0a145ff8f4d4e0484e015fb4d18860c [id=aba940f2fbb849168f3cb1bfd1c36e5e]) successfully deleted
I20250112 02:52:40.603494 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 196) took 1721 ms. Trace:
W20250112 02:52:40.603940 989 leader_election.cc:343] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 8c821ea3ca4946c290c17135366e0cac
I20250112 02:52:40.604028 989 leader_election.cc:304] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:40.604524 2326 raft_consensus.cc:2743] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:40.604588 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058a9ca089cd4967887b0a1127c7d03f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:40.604957 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a437952889494b0e820e41fdfffe708d" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:40.605109 910 raft_consensus.cc:3054] T a437952889494b0e820e41fdfffe708d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:40.604120 2562 rpcz_store.cc:276] 0112 02:52:38.882232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.882292 (+ 60us) service_pool.cc:225] Handling call
0112 02:52:40.602540 (+1720248us) ts_tablet_manager.cc:1145] Deleting tablet b9c006281dc949119951c74e23167d7f
0112 02:52:40.602833 (+ 293us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.603185 (+ 352us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1720099}
W20250112 02:52:40.606158 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet b9c006281dc949119951c74e23167d7f because the tablet was not found. No further retry: Not found: Tablet not found: b9c006281dc949119951c74e23167d7f
I20250112 02:52:40.606714 1050 raft_consensus.cc:2463] T 06edfca60eed4dd3a2dee7b662b0970c P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:40.607108 714 leader_election.cc:304] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:40.606968 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:40.608165 2451 raft_consensus.cc:2798] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:40.608294 2451 raft_consensus.cc:695] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:40.608518 2451 consensus_queue.cc:237] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.607456 910 raft_consensus.cc:2463] T a437952889494b0e820e41fdfffe708d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
W20250112 02:52:40.611025 854 leader_election.cc:343] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:40.611121 854 leader_election.cc:304] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:40.610886 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:40.611490 2696 raft_consensus.cc:2743] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:40.614454 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:40.616084 2047 consensus_queue.cc:1035] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:40.616189 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 17c54c5dc86446cf85974777c8145ef6 (table default.Table_8ab566cf9c8f45eba4dc9bb85264c823 [id=a622f7a973344a388b3275adbcce6234]). Expected version 1 got 0
I20250112 02:52:40.617429 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 92) took 6082 ms. Trace:
I20250112 02:52:40.617512 2024 rpcz_store.cc:276] 0112 02:52:34.535197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.540586 (+ 5389us) service_pool.cc:225] Handling call
0112 02:52:40.357412 (+5816826us) ts_tablet_manager.cc:1145] Deleting tablet 352c87f1dc724e7b94bc2e64abfce8d7
0112 02:52:40.357420 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.431425 (+ 74005us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.434203 (+ 2778us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.617378 (+183175us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5814337}
I20250112 02:52:40.617988 2024 tablet_replica.cc:331] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:40.618177 2024 raft_consensus.cc:2238] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:40.618307 2024 raft_consensus.cc:2267] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:40.618870 2024 ts_tablet_manager.cc:1905] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.620033 2047 consensus_queue.cc:1035] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:40.621299 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 352c87f1dc724e7b94bc2e64abfce8d7 (table default.Table_83212f303f0b4413aa6c62887da9dd34 [id=399c12e9075d4e55b21b17837ead62da]) successfully deleted
I20250112 02:52:40.624100 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:40.607787 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 198) took 1683 ms. Trace:
I20250112 02:52:40.625896 2562 rpcz_store.cc:276] 0112 02:52:38.924305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.924348 (+ 43us) service_pool.cc:225] Handling call
0112 02:52:40.607079 (+1682731us) ts_tablet_manager.cc:1145] Deleting tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b
0112 02:52:40.607090 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.607433 (+ 343us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1682253}
W20250112 02:52:40.627015 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b because the tablet was not found. No further retry: Not found: Tablet not found: 4348fe4c32704bf7b3a8d21ff3b8ac1b
I20250112 02:52:40.628142 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 201) took 1661 ms. Trace:
I20250112 02:52:40.628685 2562 rpcz_store.cc:276] 0112 02:52:38.967088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.967870 (+ 782us) service_pool.cc:225] Handling call
0112 02:52:40.627367 (+1659497us) ts_tablet_manager.cc:1145] Deleting tablet 015412a5fa304453bd8d1be1c3cd7a46
0112 02:52:40.627377 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.628119 (+ 742us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1659288}
I20250112 02:52:40.628067 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
I20250112 02:52:40.629891 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 202) took 1662 ms. Trace:
I20250112 02:52:40.630553 2562 rpcz_store.cc:276] 0112 02:52:38.967232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.968112 (+ 880us) service_pool.cc:225] Handling call
0112 02:52:40.629802 (+1661690us) ts_tablet_manager.cc:1145] Deleting tablet 64707ee432fb4e96a3bbf7252dfa1813
0112 02:52:40.629815 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.629873 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1661525}
W20250112 02:52:40.631400 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 64707ee432fb4e96a3bbf7252dfa1813 because the tablet was not found. No further retry: Not found: Tablet not found: 64707ee432fb4e96a3bbf7252dfa1813
I20250112 02:52:40.631867 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 207) took 1662 ms. Trace:
I20250112 02:52:40.632539 2562 rpcz_store.cc:276] 0112 02:52:38.969149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.969203 (+ 54us) service_pool.cc:225] Handling call
0112 02:52:40.631781 (+1662578us) ts_tablet_manager.cc:1145] Deleting tablet e5c10fad790046a88fbc8ebe0303c481
0112 02:52:40.631793 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.631849 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1662424}
W20250112 02:52:40.633283 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet e5c10fad790046a88fbc8ebe0303c481 because the tablet was not found. No further retry: Not found: Tablet not found: e5c10fad790046a88fbc8ebe0303c481
I20250112 02:52:40.633736 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 209) took 1664 ms. Trace:
I20250112 02:52:40.634230 2562 rpcz_store.cc:276] 0112 02:52:38.969456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.969499 (+ 43us) service_pool.cc:225] Handling call
0112 02:52:40.633667 (+1664168us) ts_tablet_manager.cc:1145] Deleting tablet d4abf59d96414e5faa7cf75ba133832c
0112 02:52:40.633676 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.633721 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1664013}
I20250112 02:52:40.635830 2445 raft_consensus.cc:491] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.635953 2445 raft_consensus.cc:513] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:40.636277 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet d4abf59d96414e5faa7cf75ba133832c because the tablet was not found. No further retry: Not found: Tablet not found: d4abf59d96414e5faa7cf75ba133832c
I20250112 02:52:40.636531 2445 leader_election.cc:290] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.592485 1181 raft_consensus.cc:357] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:40.637023 1181 raft_consensus.cc:383] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.637108 1181 raft_consensus.cc:738] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.637398 1181 consensus_queue.cc:260] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:40.637511 713 leader_election.cc:343] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 70715db0394d46d9b6b289160b00be7d
I20250112 02:52:40.638056 1181 ts_tablet_manager.cc:1428] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.046s user 0.001s sys 0.000s
I20250112 02:52:40.638355 1181 tablet_bootstrap.cc:492] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:40.638680 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 205) took 1669 ms. Trace:
I20250112 02:52:40.623847 2208 raft_consensus.cc:491] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.640075 2208 raft_consensus.cc:513] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.640986 2208 leader_election.cc:290] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:40.641496 991 leader_election.cc:343] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Not found: Tablet not found: b6d84027ad6d40a5be5afcdd854431a5
I20250112 02:52:40.639164 2562 rpcz_store.cc:276] 0112 02:52:38.968957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.969605 (+ 648us) service_pool.cc:225] Handling call
0112 02:52:40.638391 (+1668786us) ts_tablet_manager.cc:1145] Deleting tablet 8a12b225131b464494a9f02133d37aea
0112 02:52:40.638599 (+ 208us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.638656 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1668644}
W20250112 02:52:40.641878 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 8a12b225131b464494a9f02133d37aea because the tablet was not found. No further retry: Not found: Tablet not found: 8a12b225131b464494a9f02133d37aea
W20250112 02:52:40.642441 989 leader_election.cc:343] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: b6d84027ad6d40a5be5afcdd854431a5
I20250112 02:52:40.642581 989 leader_election.cc:304] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:40.642932 2208 raft_consensus.cc:2743] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:40.643931 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 206) took 1674 ms. Trace:
I20250112 02:52:40.645115 2562 rpcz_store.cc:276] 0112 02:52:38.969058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.969786 (+ 728us) service_pool.cc:225] Handling call
0112 02:52:40.643857 (+1674071us) ts_tablet_manager.cc:1145] Deleting tablet 5dd48d0e68014e1aa0c44620527baec9
0112 02:52:40.643867 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.643916 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1673950}
W20250112 02:52:40.646478 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 5dd48d0e68014e1aa0c44620527baec9 because the tablet was not found. No further retry: Not found: Tablet not found: 5dd48d0e68014e1aa0c44620527baec9
I20250112 02:52:40.646835 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 208) took 1677 ms. Trace:
I20250112 02:52:40.636777 1173 tablet_bootstrap.cc:492] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:40.637066 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70715db0394d46d9b6b289160b00be7d" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:40.647161 1173 ts_tablet_manager.cc:1397] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.061s user 0.014s sys 0.000s
I20250112 02:52:40.647178 1049 raft_consensus.cc:2463] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:40.647931 2562 rpcz_store.cc:276] 0112 02:52:38.969312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.969937 (+ 625us) service_pool.cc:225] Handling call
0112 02:52:40.646758 (+1676821us) ts_tablet_manager.cc:1145] Deleting tablet c4a3486ef011492c8a1df8d6278adedf
0112 02:52:40.646769 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.646820 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1676707}
I20250112 02:52:40.648003 1173 raft_consensus.cc:357] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.648162 1173 raft_consensus.cc:383] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.648217 1173 raft_consensus.cc:738] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.648458 714 leader_election.cc:304] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
W20250112 02:52:40.648772 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet c4a3486ef011492c8a1df8d6278adedf because the tablet was not found. No further retry: Not found: Tablet not found: c4a3486ef011492c8a1df8d6278adedf
I20250112 02:52:40.648906 2047 raft_consensus.cc:2798] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:40.648985 2047 raft_consensus.cc:491] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:40.649050 2047 raft_consensus.cc:3054] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:40.649909 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 203) took 1681 ms. Trace:
I20250112 02:52:40.650130 2562 rpcz_store.cc:276] 0112 02:52:38.968731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.970092 (+ 1361us) service_pool.cc:225] Handling call
0112 02:52:40.649587 (+1679495us) ts_tablet_manager.cc:1145] Deleting tablet ac716c6da01941a989754aac1ef430d5
0112 02:52:40.649599 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.649893 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1679363}
W20250112 02:52:40.650830 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet ac716c6da01941a989754aac1ef430d5 because the tablet was not found. No further retry: Not found: Tablet not found: ac716c6da01941a989754aac1ef430d5
I20250112 02:52:40.651245 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 204) took 1682 ms. Trace:
I20250112 02:52:40.651602 2562 rpcz_store.cc:276] 0112 02:52:38.968844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.970244 (+ 1400us) service_pool.cc:225] Handling call
0112 02:52:40.651161 (+1680917us) ts_tablet_manager.cc:1145] Deleting tablet f05c8426ea3d4fe89fbe6c88c688019f
0112 02:52:40.651173 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.651228 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1680771}
I20250112 02:52:40.651780 1173 consensus_queue.cc:260] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.652289 1173 ts_tablet_manager.cc:1428] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250112 02:52:40.652549 1173 tablet_bootstrap.cc:492] T bab4978edea0445fb0456e88e7fb4c44 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:40.652796 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet f05c8426ea3d4fe89fbe6c88c688019f because the tablet was not found. No further retry: Not found: Tablet not found: f05c8426ea3d4fe89fbe6c88c688019f
I20250112 02:52:40.653362 2047 raft_consensus.cc:513] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:40.654054 991 consensus_peers.cc:487] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:40.654278 2047 leader_election.cc:290] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:40.654702 713 leader_election.cc:343] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 70715db0394d46d9b6b289160b00be7d
I20250112 02:52:40.654951 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70715db0394d46d9b6b289160b00be7d" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:40.655148 1049 raft_consensus.cc:3054] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:40.658227 1049 raft_consensus.cc:2463] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:40.658579 714 leader_election.cc:304] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:40.658967 2047 raft_consensus.cc:2798] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:40.659176 2047 raft_consensus.cc:695] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:40.659416 2047 consensus_queue.cc:237] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.659454 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 212) took 1686 ms. Trace:
I20250112 02:52:40.660111 2562 rpcz_store.cc:276] 0112 02:52:38.972989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.973031 (+ 42us) service_pool.cc:225] Handling call
0112 02:52:40.659378 (+1686347us) ts_tablet_manager.cc:1145] Deleting tablet ea33e5c1a1fb46599c5836a36b130051
0112 02:52:40.659388 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.659440 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1686200}
W20250112 02:52:40.660688 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet ea33e5c1a1fb46599c5836a36b130051 because the tablet was not found. No further retry: Not found: Tablet not found: ea33e5c1a1fb46599c5836a36b130051
I20250112 02:52:40.661273 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 211) took 1688 ms. Trace:
I20250112 02:52:40.661576 2562 rpcz_store.cc:276] 0112 02:52:38.972854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.973207 (+ 353us) service_pool.cc:225] Handling call
0112 02:52:40.661198 (+1687991us) ts_tablet_manager.cc:1145] Deleting tablet 6913b0ac9ceb4dbc9cdf591f159134f7
0112 02:52:40.661207 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.661256 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1687829}
W20250112 02:52:40.662160 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 because the tablet was not found. No further retry: Not found: Tablet not found: 6913b0ac9ceb4dbc9cdf591f159134f7
I20250112 02:52:40.663018 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 214) took 1681 ms. Trace:
I20250112 02:52:40.663378 2562 rpcz_store.cc:276] 0112 02:52:38.981748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.981805 (+ 57us) service_pool.cc:225] Handling call
0112 02:52:40.662943 (+1681138us) ts_tablet_manager.cc:1145] Deleting tablet c04b64bc2f8348a086ed3c4e59f09b16
0112 02:52:40.662952 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.663002 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1680977}
I20250112 02:52:40.486547 2003 log.cc:1198] T f924459d1150472aafb325a11825db23 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/f924459d1150472aafb325a11825db23
I20250112 02:52:40.663872 2003 ts_tablet_manager.cc:1939] T f924459d1150472aafb325a11825db23 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
W20250112 02:52:40.664013 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet c04b64bc2f8348a086ed3c4e59f09b16 because the tablet was not found. No further retry: Not found: Tablet not found: c04b64bc2f8348a086ed3c4e59f09b16
I20250112 02:52:40.664330 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 215) took 1681 ms. Trace:
I20250112 02:52:40.664755 2562 rpcz_store.cc:276] 0112 02:52:38.982689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.982746 (+ 57us) service_pool.cc:225] Handling call
0112 02:52:40.664258 (+1681512us) ts_tablet_manager.cc:1145] Deleting tablet bb217afc08304c0ea35dee3120347ba1
0112 02:52:40.664268 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.664317 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1681325}
W20250112 02:52:40.665316 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet bb217afc08304c0ea35dee3120347ba1 because the tablet was not found. No further retry: Not found: Tablet not found: bb217afc08304c0ea35dee3120347ba1
I20250112 02:52:40.665704 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet f924459d1150472aafb325a11825db23 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]) successfully deleted
I20250112 02:52:40.665755 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 216) took 1682 ms. Trace:
W20250112 02:52:40.665988 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet a3ffdc237f8640f0b378809f93fc6292 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3ffdc237f8640f0b378809f93fc6292 already in progress: creating tablet
I20250112 02:52:40.666111 2562 rpcz_store.cc:276] 0112 02:52:38.983457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.983521 (+ 64us) service_pool.cc:225] Handling call
0112 02:52:40.665682 (+1682161us) ts_tablet_manager.cc:1145] Deleting tablet 4b599e983ec04f53a440d1ab828d12ab
0112 02:52:40.665692 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.665741 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1681996}
W20250112 02:52:40.666196 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f3771166b77e4f90aaa116ab1a9dc299 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3771166b77e4f90aaa116ab1a9dc299 already in progress: creating tablet
W20250112 02:52:40.666424 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d63d411f906b4d4eb68343bdcc493e05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d63d411f906b4d4eb68343bdcc493e05 already in progress: creating tablet
W20250112 02:52:40.666608 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 7e1954e7859f422f9690fe4a9384a8bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e1954e7859f422f9690fe4a9384a8bb already in progress: creating tablet
W20250112 02:52:40.666702 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 4b599e983ec04f53a440d1ab828d12ab because the tablet was not found. No further retry: Not found: Tablet not found: 4b599e983ec04f53a440d1ab828d12ab
W20250112 02:52:40.666832 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet bab4978edea0445fb0456e88e7fb4c44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bab4978edea0445fb0456e88e7fb4c44 already in progress: creating tablet
W20250112 02:52:40.667029 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 8c821ea3ca4946c290c17135366e0cac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c821ea3ca4946c290c17135366e0cac already in progress: creating tablet
I20250112 02:52:40.667063 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 217) took 1682 ms. Trace:
W20250112 02:52:40.667294 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f234256b343f447dadfc3d1936a824ee because the tablet was not found. No further retry: Not found: Tablet not found: f234256b343f447dadfc3d1936a824ee
W20250112 02:52:40.667480 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b9c006281dc949119951c74e23167d7f because the tablet was not found. No further retry: Not found: Tablet not found: b9c006281dc949119951c74e23167d7f
I20250112 02:52:40.667398 2562 rpcz_store.cc:276] 0112 02:52:38.984277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.984327 (+ 50us) service_pool.cc:225] Handling call
0112 02:52:40.666991 (+1682664us) ts_tablet_manager.cc:1145] Deleting tablet a85300a9aa1c4c50a3957a004bea7933
0112 02:52:40.666999 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.667049 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1682505}
W20250112 02:52:40.667717 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f05c8426ea3d4fe89fbe6c88c688019f because the tablet was not found. No further retry: Not found: Tablet not found: f05c8426ea3d4fe89fbe6c88c688019f
W20250112 02:52:40.667932 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet c7c530aa0aa64e47ae27d61f341d2bd9 because the tablet was not found. No further retry: Not found: Tablet not found: c7c530aa0aa64e47ae27d61f341d2bd9
W20250112 02:52:40.668134 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 8ee98f055024429eafaa29bb961cab87 because the tablet was not found. No further retry: Not found: Tablet not found: 8ee98f055024429eafaa29bb961cab87
W20250112 02:52:40.668211 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet a85300a9aa1c4c50a3957a004bea7933 because the tablet was not found. No further retry: Not found: Tablet not found: a85300a9aa1c4c50a3957a004bea7933
W20250112 02:52:40.668365 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 82b351c563d749df806ab5d0331a4ef2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82b351c563d749df806ab5d0331a4ef2 already in progress: creating tablet
I20250112 02:52:40.668579 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 223) took 1643 ms. Trace:
I20250112 02:52:40.668838 2562 rpcz_store.cc:276] 0112 02:52:39.025352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.025396 (+ 44us) service_pool.cc:225] Handling call
0112 02:52:40.668509 (+1643113us) ts_tablet_manager.cc:1145] Deleting tablet a12c5d58e6c246c983e8a533906ce77d
0112 02:52:40.668518 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.668566 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1642966}
W20250112 02:52:40.669425 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet a12c5d58e6c246c983e8a533906ce77d because the tablet was not found. No further retry: Not found: Tablet not found: a12c5d58e6c246c983e8a533906ce77d
I20250112 02:52:40.669819 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 225) took 1644 ms. Trace:
I20250112 02:52:40.670156 2562 rpcz_store.cc:276] 0112 02:52:39.025561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.025594 (+ 33us) service_pool.cc:225] Handling call
0112 02:52:40.669730 (+1644136us) ts_tablet_manager.cc:1145] Deleting tablet b8daf4ef3593410eae87672402292be6
0112 02:52:40.669740 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.669800 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1644023}
W20250112 02:52:40.670799 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet b8daf4ef3593410eae87672402292be6 because the tablet was not found. No further retry: Not found: Tablet not found: b8daf4ef3593410eae87672402292be6
I20250112 02:52:40.671422 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 227) took 1645 ms. Trace:
W20250112 02:52:40.671865 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 62f4bd2df0ce45dabc55373f1154d3ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62f4bd2df0ce45dabc55373f1154d3ac already in progress: creating tablet
I20250112 02:52:40.671830 2562 rpcz_store.cc:276] 0112 02:52:39.025746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.025786 (+ 40us) service_pool.cc:225] Handling call
0112 02:52:40.671333 (+1645547us) ts_tablet_manager.cc:1145] Deleting tablet f234256b343f447dadfc3d1936a824ee
0112 02:52:40.671344 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.671405 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1645427}
W20250112 02:52:40.672528 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet f234256b343f447dadfc3d1936a824ee because the tablet was not found. No further retry: Not found: Tablet not found: f234256b343f447dadfc3d1936a824ee
I20250112 02:52:40.672869 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 218) took 1648 ms. Trace:
I20250112 02:52:40.673166 2562 rpcz_store.cc:276] 0112 02:52:39.024765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.025813 (+ 1048us) service_pool.cc:225] Handling call
0112 02:52:40.672800 (+1646987us) ts_tablet_manager.cc:1145] Deleting tablet 9d5b3df88a0c45dba879b3d506710614
0112 02:52:40.672807 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.672854 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1646845}
W20250112 02:52:40.673804 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 9d5b3df88a0c45dba879b3d506710614 because the tablet was not found. No further retry: Not found: Tablet not found: 9d5b3df88a0c45dba879b3d506710614
W20250112 02:52:40.674047 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet a8e7882e07784bdda18dc1ffd20963c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8e7882e07784bdda18dc1ffd20963c8 already in progress: creating tablet
I20250112 02:52:40.674312 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 219) took 1649 ms. Trace:
I20250112 02:52:40.674594 2562 rpcz_store.cc:276] 0112 02:52:39.024921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.026005 (+ 1084us) service_pool.cc:225] Handling call
0112 02:52:40.674103 (+1648098us) ts_tablet_manager.cc:1145] Deleting tablet 1b184806b4884cd3b0a002905a1e57af
0112 02:52:40.674227 (+ 124us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.674294 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1647979}
I20250112 02:52:40.674538 2003 tablet_replica.cc:331] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:40.675305 2024 ts_tablet_manager.cc:1918] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 02:52:40.675400 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 1b184806b4884cd3b0a002905a1e57af because the tablet was not found. No further retry: Not found: Tablet not found: 1b184806b4884cd3b0a002905a1e57af
I20250112 02:52:40.675828 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 220) took 1650 ms. Trace:
I20250112 02:52:40.676208 2562 rpcz_store.cc:276] 0112 02:52:39.025073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.026144 (+ 1071us) service_pool.cc:225] Handling call
0112 02:52:40.675730 (+1649586us) ts_tablet_manager.cc:1145] Deleting tablet 352c87f1dc724e7b94bc2e64abfce8d7
0112 02:52:40.675754 (+ 24us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.675814 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1649471}
I20250112 02:52:40.675817 2003 raft_consensus.cc:2238] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
W20250112 02:52:40.676862 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 352c87f1dc724e7b94bc2e64abfce8d7 because the tablet was not found. No further retry: Not found: Tablet not found: 352c87f1dc724e7b94bc2e64abfce8d7
I20250112 02:52:40.677253 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 221) took 1652 ms. Trace:
I20250112 02:52:40.677407 2003 pending_rounds.cc:62] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:40.677595 2562 rpcz_store.cc:276] 0112 02:52:39.025198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.026277 (+ 1079us) service_pool.cc:225] Handling call
0112 02:52:40.677169 (+1650892us) ts_tablet_manager.cc:1145] Deleting tablet 73136be25c79491784d3ea71aa9b27f2
0112 02:52:40.677180 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.677236 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1650797}
W20250112 02:52:40.681617 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 73136be25c79491784d3ea71aa9b27f2 because the tablet was not found. No further retry: Not found: Tablet not found: 73136be25c79491784d3ea71aa9b27f2
I20250112 02:52:40.681993 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 222) took 1656 ms. Trace:
I20250112 02:52:40.682133 2562 rpcz_store.cc:276] 0112 02:52:39.025283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.026384 (+ 1101us) service_pool.cc:225] Handling call
0112 02:52:40.681902 (+1655518us) ts_tablet_manager.cc:1145] Deleting tablet 6cb76746b3a846c3aba531c04bf63966
0112 02:52:40.681913 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.681976 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1655428}
W20250112 02:52:40.682631 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 6cb76746b3a846c3aba531c04bf63966 because the tablet was not found. No further retry: Not found: Tablet not found: 6cb76746b3a846c3aba531c04bf63966
I20250112 02:52:40.682996 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 224) took 1657 ms. Trace:
I20250112 02:52:40.683154 2562 rpcz_store.cc:276] 0112 02:52:39.025480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.026513 (+ 1033us) service_pool.cc:225] Handling call
0112 02:52:40.682906 (+1656393us) ts_tablet_manager.cc:1145] Deleting tablet c8a7281c0c09442ebd0cc13b8496c86e
0112 02:52:40.682919 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.682978 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1656236}
W20250112 02:52:40.683673 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet c8a7281c0c09442ebd0cc13b8496c86e because the tablet was not found. No further retry: Not found: Tablet not found: c8a7281c0c09442ebd0cc13b8496c86e
I20250112 02:52:40.683956 2562 tablet_replica.cc:331] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:40.684165 2562 raft_consensus.cc:2238] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:40.684386 2562 raft_consensus.cc:2267] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:40.680384 1181 tablet_bootstrap.cc:654] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.685751 749 tablet_service.cc:1467] Processing CreateTablet for tablet be856e4b913e40d58a3f3ba54698cc5f (DEFAULT_TABLE table=default.Table_82731e14956e49edb4d2103794e88fec [id=9a8fe3b96ed34c95814c9c19b8804e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.686120 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be856e4b913e40d58a3f3ba54698cc5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.686846 2562 ts_tablet_manager.cc:1905] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.677752 2003 pending_rounds.cc:69] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319877054771200 op_type: NO_OP noop_request { }
I20250112 02:52:40.689785 2003 raft_consensus.cc:2883] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:40.690325 2003 raft_consensus.cc:2267] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:40.693578 897 tablet_service.cc:1467] Processing CreateTablet for tablet be856e4b913e40d58a3f3ba54698cc5f (DEFAULT_TABLE table=default.Table_82731e14956e49edb4d2103794e88fec [id=9a8fe3b96ed34c95814c9c19b8804e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.694020 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be856e4b913e40d58a3f3ba54698cc5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.694908 2003 ts_tablet_manager.cc:1905] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.695199 749 tablet_service.cc:1467] Processing CreateTablet for tablet fc2bc28710d047cf92f6e92509c2bbaf (DEFAULT_TABLE table=default.Table_56415126d9a642c386bd4058b011e2b5 [id=4ffae95dc66443e599b8cc75338b45f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.696709 747 tablet_service.cc:1467] Processing CreateTablet for tablet 3dab0466476c4a9aa265271c1abd91a4 (DEFAULT_TABLE table=default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.697022 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dab0466476c4a9aa265271c1abd91a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.699014 890 tablet_service.cc:1467] Processing CreateTablet for tablet fc2bc28710d047cf92f6e92509c2bbaf (DEFAULT_TABLE table=default.Table_56415126d9a642c386bd4058b011e2b5 [id=4ffae95dc66443e599b8cc75338b45f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.699383 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc2bc28710d047cf92f6e92509c2bbaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.701469 760 tablet_service.cc:1467] Processing CreateTablet for tablet 9706187d3f9d4368a8e47962bc73ef3d (DEFAULT_TABLE table=default.Table_0182c19fa6214ea1928001cb0359f262 [id=21a1b4cc762f45f8b6c6327c992584da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.701764 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9706187d3f9d4368a8e47962bc73ef3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.703114 2696 raft_consensus.cc:491] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.703218 2696 raft_consensus.cc:513] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:40.703758 2696 leader_election.cc:290] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:40.704691 854 leader_election.cc:343] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 5804fc4130f84c69832c1022b7bd7fa5
I20250112 02:52:40.706238 899 tablet_service.cc:1467] Processing CreateTablet for tablet 3dab0466476c4a9aa265271c1abd91a4 (DEFAULT_TABLE table=default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.706573 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dab0466476c4a9aa265271c1abd91a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.709414 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc2bc28710d047cf92f6e92509c2bbaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.710500 895 tablet_service.cc:1467] Processing CreateTablet for tablet 9706187d3f9d4368a8e47962bc73ef3d (DEFAULT_TABLE table=default.Table_0182c19fa6214ea1928001cb0359f262 [id=21a1b4cc762f45f8b6c6327c992584da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.710853 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9706187d3f9d4368a8e47962bc73ef3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.713045 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:40.713292 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:40.713425 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
I20250112 02:52:40.713634 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:40.714102 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:40.714293 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 17c54c5dc86446cf85974777c8145ef6 (table default.Table_8ab566cf9c8f45eba4dc9bb85264c823 [id=a622f7a973344a388b3275adbcce6234]). Expected version 1 got 0
I20250112 02:52:40.714432 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:40.714597 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:40.715114 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:40.715374 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5cc414d6420c41839621ab858067548b (table default.Table_8fe6c818f15341e08fe8870664eac4c1 [id=268dae304c9143a881c7a0898632a3e4]). Expected version 2 got 1
I20250112 02:52:40.715663 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:40.715690 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5804fc4130f84c69832c1022b7bd7fa5" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:40.715881 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:40.716140 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
W20250112 02:52:40.716183 855 leader_election.cc:343] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:40.716267 855 leader_election.cc:304] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:40.716478 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:40.716847 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:40.716984 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:40.717113 2696 raft_consensus.cc:2743] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:40.717483 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:40.717875 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:40.718024 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:40.718365 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:40.718519 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:40.718814 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ca2165362ee541adac724b63f39b1489 (table default.Table_41b7056b6e9240928d0af52aec2e9338 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 2 got 0
I20250112 02:52:40.719173 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 1 got 0
I20250112 02:52:40.730441 2003 ts_tablet_manager.cc:1918] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:40.731248 2003 log.cc:1198] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/06edfca60eed4dd3a2dee7b662b0970c
I20250112 02:52:40.733120 2003 ts_tablet_manager.cc:1939] T 06edfca60eed4dd3a2dee7b662b0970c P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:40.719431 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:40.734267 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_a9907ed8bfe14625977ec98b67e9d599 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 2 got 0
I20250112 02:52:40.735913 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 06edfca60eed4dd3a2dee7b662b0970c (table default.Table_2a2995d7964346c08fcbff96b4cc3ca5 [id=70371e156f2f401ab5ee0e4e4c7545b9]) successfully deleted
W20250112 02:52:40.736145 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 7a34d03a80ae402db6607261a5c40b60 because the tablet was not found. No further retry: Not found: Tablet not found: 7a34d03a80ae402db6607261a5c40b60
W20250112 02:52:40.736953 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet cbe30f9860e2492b90e8cedf8ff9daba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbe30f9860e2492b90e8cedf8ff9daba already in progress: creating tablet
W20250112 02:52:40.737977 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet a12c5d58e6c246c983e8a533906ce77d because the tablet was not found. No further retry: Not found: Tablet not found: a12c5d58e6c246c983e8a533906ce77d
W20250112 02:52:40.738615 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet c29a9ca43e034b43a00207e5847bbee3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c29a9ca43e034b43a00207e5847bbee3 already in progress: creating tablet
I20250112 02:52:40.737807 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:40.739022 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:40.739228 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_5d473cabff2e46e0b531fbce28d9d558 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 1 got 0
W20250112 02:52:40.739696 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 54215c55966a46e1ab96e352c4e3764e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54215c55966a46e1ab96e352c4e3764e already in progress: creating tablet
I20250112 02:52:40.740388 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:40.741058 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:40.741420 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:40.742913 2003 tablet_replica.cc:331] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
W20250112 02:52:40.744500 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet e5c10fad790046a88fbc8ebe0303c481 because the tablet was not found. No further retry: Not found: Tablet not found: e5c10fad790046a88fbc8ebe0303c481
W20250112 02:52:40.744709 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f776e2a6ec704cf9b64483bc7d5b1dff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f776e2a6ec704cf9b64483bc7d5b1dff already in progress: creating tablet
W20250112 02:52:40.744940 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet fe614eaeb793473d9b2da206b7345562 because the tablet was not found. No further retry: Not found: Tablet not found: fe614eaeb793473d9b2da206b7345562
W20250112 02:52:40.745151 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 13578542e9e741e8817e0ee63b10f1a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13578542e9e741e8817e0ee63b10f1a1 already in progress: creating tablet
I20250112 02:52:40.750373 2003 raft_consensus.cc:2238] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:40.751086 2003 pending_rounds.cc:62] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:40.751212 2003 pending_rounds.cc:69] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319877263306752 op_type: NO_OP noop_request { }
I20250112 02:52:40.751377 2003 raft_consensus.cc:2883] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:40.751554 2003 raft_consensus.cc:2267] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:40.759552 2003 ts_tablet_manager.cc:1905] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.767406 1177 tablet_bootstrap.cc:492] T 46d1beac4410422bbf1e912d35999f3d P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:40.767581 1177 ts_tablet_manager.cc:1397] T 46d1beac4410422bbf1e912d35999f3d P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.250s user 0.018s sys 0.044s
I20250112 02:52:40.768460 1177 raft_consensus.cc:357] T 46d1beac4410422bbf1e912d35999f3d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.768692 1177 raft_consensus.cc:383] T 46d1beac4410422bbf1e912d35999f3d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.768760 1177 raft_consensus.cc:738] T 46d1beac4410422bbf1e912d35999f3d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.769052 2024 log.cc:1198] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/b9c006281dc949119951c74e23167d7f
I20250112 02:52:40.768992 1177 consensus_queue.cc:260] T 46d1beac4410422bbf1e912d35999f3d P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.769361 2609 raft_consensus.cc:491] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.769404 1177 ts_tablet_manager.cc:1428] T 46d1beac4410422bbf1e912d35999f3d P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 02:52:40.769524 2609 raft_consensus.cc:513] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.769631 1177 tablet_bootstrap.cc:492] T 8c821ea3ca4946c290c17135366e0cac P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:40.770087 2609 leader_election.cc:290] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.689229 1031 tablet_service.cc:1467] Processing CreateTablet for tablet be856e4b913e40d58a3f3ba54698cc5f (DEFAULT_TABLE table=default.Table_82731e14956e49edb4d2103794e88fec [id=9a8fe3b96ed34c95814c9c19b8804e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.770587 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be856e4b913e40d58a3f3ba54698cc5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.772764 2024 ts_tablet_manager.cc:1939] T b9c006281dc949119951c74e23167d7f P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:40.773056 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 102) took 6235 ms. Trace:
I20250112 02:52:40.773135 2024 rpcz_store.cc:276] 0112 02:52:34.537244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.541645 (+ 4401us) service_pool.cc:225] Handling call
0112 02:52:40.617782 (+6076137us) ts_tablet_manager.cc:1145] Deleting tablet b9c006281dc949119951c74e23167d7f
0112 02:52:40.617791 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.672176 (+ 54385us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.675229 (+ 3053us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.773041 (+ 97812us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1765,"tablet-delete.queue_time_us":6075993}
I20250112 02:52:40.774109 1177 tablet_bootstrap.cc:654] T 8c821ea3ca4946c290c17135366e0cac P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.773468 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 110) took 5915 ms. Trace:
I20250112 02:52:40.776376 2024 rpcz_store.cc:276] 0112 02:52:34.857718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.857772 (+ 54us) service_pool.cc:225] Handling call
0112 02:52:40.773363 (+5915591us) ts_tablet_manager.cc:1145] Deleting tablet a8e7882e07784bdda18dc1ffd20963c8
0112 02:52:40.773371 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.773456 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5915338}
I20250112 02:52:40.776726 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 113) took 5910 ms. Trace:
I20250112 02:52:40.778532 2003 ts_tablet_manager.cc:1918] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:40.779183 2003 log.cc:1198] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/70715db0394d46d9b6b289160b00be7d
I20250112 02:52:40.781365 2003 ts_tablet_manager.cc:1939] T 70715db0394d46d9b6b289160b00be7d P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:40.783944 2024 rpcz_store.cc:276] 0112 02:52:34.866720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.866777 (+ 57us) service_pool.cc:225] Handling call
0112 02:52:40.776634 (+5909857us) ts_tablet_manager.cc:1145] Deleting tablet c29a9ca43e034b43a00207e5847bbee3
0112 02:52:40.776642 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.776712 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5909609}
I20250112 02:52:40.784287 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 114) took 5917 ms. Trace:
I20250112 02:52:40.784377 2024 rpcz_store.cc:276] 0112 02:52:34.866869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.867055 (+ 186us) service_pool.cc:225] Handling call
0112 02:52:40.784201 (+5917146us) ts_tablet_manager.cc:1145] Deleting tablet 83c04f1a2cb04950a9f6170e252c19b4
0112 02:52:40.784208 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.784274 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5917011}
I20250112 02:52:40.785604 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 70715db0394d46d9b6b289160b00be7d (table default.Table_bc3feb9849cf4e5db05a15330115dd52 [id=53797be8d53c45f3b805e532889be30c]) successfully deleted
W20250112 02:52:40.786836 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet c4a3486ef011492c8a1df8d6278adedf because the tablet was not found. No further retry: Not found: Tablet not found: c4a3486ef011492c8a1df8d6278adedf
W20250112 02:52:40.787830 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 83c04f1a2cb04950a9f6170e252c19b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83c04f1a2cb04950a9f6170e252c19b4 already in progress: creating tablet
I20250112 02:52:40.788924 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 109) took 5931 ms. Trace:
I20250112 02:52:40.789037 2024 rpcz_store.cc:276] 0112 02:52:34.857589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.876062 (+ 18473us) service_pool.cc:225] Handling call
0112 02:52:40.788828 (+5912766us) ts_tablet_manager.cc:1145] Deleting tablet 62f4bd2df0ce45dabc55373f1154d3ac
0112 02:52:40.788838 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.788910 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5912474}
W20250112 02:52:40.789268 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet bd5b3c712c0f448ea9b14ab93dd731ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd5b3c712c0f448ea9b14ab93dd731ab already in progress: creating tablet
I20250112 02:52:40.694914 1173 tablet_bootstrap.cc:654] T bab4978edea0445fb0456e88e7fb4c44 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.695780 1017 tablet_service.cc:1467] Processing CreateTablet for tablet fc2bc28710d047cf92f6e92509c2bbaf (DEFAULT_TABLE table=default.Table_56415126d9a642c386bd4058b011e2b5 [id=4ffae95dc66443e599b8cc75338b45f0]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:40.789789 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 87759e8b99b04e628badc8f60cb38bb3 because the tablet was not found. No further retry: Not found: Tablet not found: 87759e8b99b04e628badc8f60cb38bb3
I20250112 02:52:40.699666 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 3dab0466476c4a9aa265271c1abd91a4 (DEFAULT_TABLE table=default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.704223 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 9706187d3f9d4368a8e47962bc73ef3d (DEFAULT_TABLE table=default.Table_0182c19fa6214ea1928001cb0359f262 [id=21a1b4cc762f45f8b6c6327c992584da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.790072 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc2bc28710d047cf92f6e92509c2bbaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.735594 2208 consensus_queue.cc:1035] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:40.790807 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9706187d3f9d4368a8e47962bc73ef3d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:40.745339 989 consensus_peers.cc:487] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4348fe4c32704bf7b3a8d21ff3b8ac1b. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:52:40.791270 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 45957.2us
I20250112 02:52:40.792870 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet b9c006281dc949119951c74e23167d7f (table default.Table_1a6ba73a53b14addbacc6a2ff0711256 [id=fe95e997b21346a5b36675a9c4d25a06]) successfully deleted
W20250112 02:52:40.793087 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet a8e7882e07784bdda18dc1ffd20963c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8e7882e07784bdda18dc1ffd20963c8 already in progress: creating tablet
W20250112 02:52:40.793372 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet c29a9ca43e034b43a00207e5847bbee3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c29a9ca43e034b43a00207e5847bbee3 already in progress: creating tablet
W20250112 02:52:40.793566 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 83c04f1a2cb04950a9f6170e252c19b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83c04f1a2cb04950a9f6170e252c19b4 already in progress: creating tablet
W20250112 02:52:40.793735 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 62f4bd2df0ce45dabc55373f1154d3ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62f4bd2df0ce45dabc55373f1154d3ac already in progress: creating tablet
I20250112 02:52:40.794901 749 tablet_service.cc:1467] Processing CreateTablet for tablet bd6272f9d0754f0cbc1fb78391f0f927 (DEFAULT_TABLE table=default.Table_8bf5e22d7ca64a46b04eecfa87d1a646 [id=974915ca8d414c1ca5f5aa76dcf81d1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.795653 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dab0466476c4a9aa265271c1abd91a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.800489 1017 tablet_service.cc:1467] Processing CreateTablet for tablet bd6272f9d0754f0cbc1fb78391f0f927 (DEFAULT_TABLE table=default.Table_8bf5e22d7ca64a46b04eecfa87d1a646 [id=974915ca8d414c1ca5f5aa76dcf81d1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.800776 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6272f9d0754f0cbc1fb78391f0f927. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.803086 2208 consensus_queue.cc:1035] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:40.804018 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42444e99aba84a7b873bc8cdf954a6dc" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:40.804239 918 raft_consensus.cc:2463] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:40.804612 989 leader_election.cc:304] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:40.804814 2208 raft_consensus.cc:2798] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:40.804903 2208 raft_consensus.cc:491] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:40.804960 2208 raft_consensus.cc:3054] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:40.807402 2208 raft_consensus.cc:513] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.810070 2208 leader_election.cc:290] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.811545 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42444e99aba84a7b873bc8cdf954a6dc" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:40.811698 918 raft_consensus.cc:3054] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:40.811869 760 tablet_service.cc:1467] Processing CreateTablet for tablet f60626faf203441ab48c60147ddc3cf8 (DEFAULT_TABLE table=default.Table_7f1cd8fa7b844d0fba0cad8d1a6652e6 [id=cc8ce7ac6e4a416486ad1175ca1e1d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.812588 747 tablet_service.cc:1467] Processing CreateTablet for tablet 262b5fa574fd4baa83aed9d1cf08cdca (DEFAULT_TABLE table=default.Table_0063877a6c23462bb1550693378db102 [id=660352adb99642338d9ce4621bb08912]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.815038 918 raft_consensus.cc:2463] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:40.815407 989 leader_election.cc:304] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:40.815546 895 tablet_service.cc:1467] Processing CreateTablet for tablet f60626faf203441ab48c60147ddc3cf8 (DEFAULT_TABLE table=default.Table_7f1cd8fa7b844d0fba0cad8d1a6652e6 [id=cc8ce7ac6e4a416486ad1175ca1e1d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.815977 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f60626faf203441ab48c60147ddc3cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.817924 2208 raft_consensus.cc:2798] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:40.818066 2208 raft_consensus.cc:695] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:40.818342 2208 consensus_queue.cc:237] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.818907 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:40.820655 743 tablet_service.cc:1467] Processing CreateTablet for tablet 1164804f2e2644b589ff727f499ed6a1 (DEFAULT_TABLE table=default.Table_38abdcafa8f94c1a82760c1c48f93768 [id=b915ff23c73e49d7a3d983931ddfad03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.820747 890 tablet_service.cc:1467] Processing CreateTablet for tablet 262b5fa574fd4baa83aed9d1cf08cdca (DEFAULT_TABLE table=default.Table_0063877a6c23462bb1550693378db102 [id=660352adb99642338d9ce4621bb08912]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.821048 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 262b5fa574fd4baa83aed9d1cf08cdca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.821440 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 262b5fa574fd4baa83aed9d1cf08cdca (DEFAULT_TABLE table=default.Table_0063877a6c23462bb1550693378db102 [id=660352adb99642338d9ce4621bb08912]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.822458 744 tablet_service.cc:1467] Processing CreateTablet for tablet 8c12a1c5e3584450ac193743a0107e80 (DEFAULT_TABLE table=default.Table_22969d80f802435898d7d60983f0daf9 [id=5341267f9577457697f7768c6489768c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.822742 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 262b5fa574fd4baa83aed9d1cf08cdca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.789322 2024 tablet_replica.cc:331] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:40.823520 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 8c12a1c5e3584450ac193743a0107e80 (DEFAULT_TABLE table=default.Table_22969d80f802435898d7d60983f0daf9 [id=5341267f9577457697f7768c6489768c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.823951 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c12a1c5e3584450ac193743a0107e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.825870 899 tablet_service.cc:1467] Processing CreateTablet for tablet 1164804f2e2644b589ff727f499ed6a1 (DEFAULT_TABLE table=default.Table_38abdcafa8f94c1a82760c1c48f93768 [id=b915ff23c73e49d7a3d983931ddfad03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.826210 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1164804f2e2644b589ff727f499ed6a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.828481 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 1164804f2e2644b589ff727f499ed6a1 (DEFAULT_TABLE table=default.Table_38abdcafa8f94c1a82760c1c48f93768 [id=b915ff23c73e49d7a3d983931ddfad03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.828614 2024 raft_consensus.cc:2238] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:40.828810 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1164804f2e2644b589ff727f499ed6a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.828995 2024 raft_consensus.cc:2267] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:40.829588 898 tablet_service.cc:1467] Processing CreateTablet for tablet 8c12a1c5e3584450ac193743a0107e80 (DEFAULT_TABLE table=default.Table_22969d80f802435898d7d60983f0daf9 [id=5341267f9577457697f7768c6489768c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.829753 2024 ts_tablet_manager.cc:1905] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.829835 756 tablet_service.cc:1467] Processing CreateTablet for tablet 64b0d58af4cf4adcbeb3ef5070f2982d (DEFAULT_TABLE table=default.Table_bb82d19aab034576ad7a0303219fa410 [id=e1067ea6372e45e0b11b8e5713cd71cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.829926 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c12a1c5e3584450ac193743a0107e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.830595 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 64b0d58af4cf4adcbeb3ef5070f2982d (DEFAULT_TABLE table=default.Table_bb82d19aab034576ad7a0303219fa410 [id=e1067ea6372e45e0b11b8e5713cd71cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.830938 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64b0d58af4cf4adcbeb3ef5070f2982d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.836057 897 tablet_service.cc:1467] Processing CreateTablet for tablet bd6272f9d0754f0cbc1fb78391f0f927 (DEFAULT_TABLE table=default.Table_8bf5e22d7ca64a46b04eecfa87d1a646 [id=974915ca8d414c1ca5f5aa76dcf81d1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.836401 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6272f9d0754f0cbc1fb78391f0f927. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.839510 754 tablet_service.cc:1467] Processing CreateTablet for tablet 574de2480cbb4498b414c284fea30b2c (DEFAULT_TABLE table=default.Table_e7153c9aa3634845b04fa8458133877c [id=e355bf6e21ea4e2f8d2f14bae71f7fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.839715 898 tablet_service.cc:1467] Processing CreateTablet for tablet 64b0d58af4cf4adcbeb3ef5070f2982d (DEFAULT_TABLE table=default.Table_bb82d19aab034576ad7a0303219fa410 [id=e1067ea6372e45e0b11b8e5713cd71cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.840013 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 574de2480cbb4498b414c284fea30b2c (DEFAULT_TABLE table=default.Table_e7153c9aa3634845b04fa8458133877c [id=e355bf6e21ea4e2f8d2f14bae71f7fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.840291 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 574de2480cbb4498b414c284fea30b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.840291 889 tablet_service.cc:1467] Processing CreateTablet for tablet 574de2480cbb4498b414c284fea30b2c (DEFAULT_TABLE table=default.Table_e7153c9aa3634845b04fa8458133877c [id=e355bf6e21ea4e2f8d2f14bae71f7fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.840852 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 574de2480cbb4498b414c284fea30b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.843124 1035 tablet_service.cc:1467] Processing CreateTablet for tablet f60626faf203441ab48c60147ddc3cf8 (DEFAULT_TABLE table=default.Table_7f1cd8fa7b844d0fba0cad8d1a6652e6 [id=cc8ce7ac6e4a416486ad1175ca1e1d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:40.846304 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64b0d58af4cf4adcbeb3ef5070f2982d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.843456 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f60626faf203441ab48c60147ddc3cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.849133 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:40.849766 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:40.849982 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:40.850140 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:40.850349 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_5d473cabff2e46e0b531fbce28d9d558 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 1 got 0
I20250112 02:52:40.850525 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:40.850699 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_a9907ed8bfe14625977ec98b67e9d599 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 2 got 0
I20250112 02:52:40.851099 453 catalog_manager.cc:5526] T 4fc8658b0efe43bdb339edbff64973f6 P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:40.852350 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 1 got 0
I20250112 02:52:40.852669 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ca2165362ee541adac724b63f39b1489 (table default.Table_41b7056b6e9240928d0af52aec2e9338 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 2 got 0
I20250112 02:52:40.853091 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:40.853349 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:40.854872 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:40.855679 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:40.855901 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:40.856488 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:40.856878 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:40.857419 453 catalog_manager.cc:5526] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:40.857878 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
I20250112 02:52:40.858125 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:40.858345 2003 tablet_replica.cc:331] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:40.858470 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:40.858525 2003 raft_consensus.cc:2238] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:40.858628 453 catalog_manager.cc:5526] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:40.858808 2003 raft_consensus.cc:2267] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:40.859159 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6272f9d0754f0cbc1fb78391f0f927. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.859176 2208 raft_consensus.cc:491] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.859220 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:40.859294 2208 raft_consensus.cc:513] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:40.859405 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:40.859835 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:40.859874 2208 leader_election.cc:290] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.860212 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:40.860230 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a0d939fdb94590a86389f1f4693e9a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:40.860365 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 17c54c5dc86446cf85974777c8145ef6 (table default.Table_8ab566cf9c8f45eba4dc9bb85264c823 [id=a622f7a973344a388b3275adbcce6234]). Expected version 1 got 0
I20250112 02:52:40.860385 918 raft_consensus.cc:2463] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:40.860673 989 leader_election.cc:304] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:40.860814 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:40.861261 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:40.861421 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
I20250112 02:52:40.861586 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:40.861799 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:40.861961 2208 raft_consensus.cc:2798] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:40.862038 2208 raft_consensus.cc:491] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:40.862102 2208 raft_consensus.cc:3054] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:40.862393 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d95f98ae7bdf42518bbff55e257fea88 consensus state has no leader
I20250112 02:52:40.862561 449 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 165) took 1402 ms. Trace:
I20250112 02:52:40.862634 449 rpcz_store.cc:276] 0112 02:52:39.460233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.378710 (+918477us) service_pool.cc:225] Handling call
0112 02:52:40.387605 (+ 8895us) spinlock_profiling.cc:243] Waited 8.63 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd74f96 00007f3e13e78bb6 00007f3e13e788e5 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:40.387879 (+ 274us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:40.388087 (+ 208us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:40.388233 (+ 146us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:40.388304 (+ 71us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:40.475175 (+ 86871us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:40.862516 (+387341us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:40.388677 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.388766 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.388770 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:40.388772 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:40.388853 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.388862 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.388866 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:40.388920 (+ 54us) tablet.cc:698] Row locks acquired
0112 02:52:40.388922 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:40.388949 (+ 27us) write_op.cc:270] Start()
0112 02:52:40.388983 (+ 34us) write_op.cc:276] Timestamp: P: 1736650360388887 usec, L: 0
0112 02:52:40.388988 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.389103 (+ 115us) log.cc:844] Serialized 405 byte log entry
0112 02:52:40.473460 (+ 84357us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.474112 (+ 652us) write_op.cc:301] APPLY: starting
0112 02:52:40.474121 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.474156 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.474173 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.474203 (+ 30us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.474225 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:40.474230 (+ 5us) write_op.cc:312] APPLY: finished
0112 02:52:40.474277 (+ 47us) log.cc:844] Serialized 23 byte log entry
0112 02:52:40.474311 (+ 34us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.474321 (+ 10us) write_op.cc:454] Released schema lock
0112 02:52:40.474325 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:40.474381 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.474387 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":19857280,"child_traces":[["op",{"apply.queue_time_us":621,"apply.run_cpu_time_us":261,"apply.run_wall_time_us":262,"num_ops":1,"prepare.queue_time_us":41,"prepare.run_cpu_time_us":475,"prepare.run_wall_time_us":475,"replication_time_us":84452}]]}
I20250112 02:52:40.865031 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f60626faf203441ab48c60147ddc3cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.867693 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 262b5fa574fd4baa83aed9d1cf08cdca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.870386 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1164804f2e2644b589ff727f499ed6a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.873098 2208 raft_consensus.cc:513] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:40.874084 2003 ts_tablet_manager.cc:1905] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.875679 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c12a1c5e3584450ac193743a0107e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.879251 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64b0d58af4cf4adcbeb3ef5070f2982d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:40.883667 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 574de2480cbb4498b414c284fea30b2c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:40.887701 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 416fb6820cd54e118950c68d381f51b2 because the tablet was not found. No further retry: Not found: Tablet not found: 416fb6820cd54e118950c68d381f51b2
W20250112 02:52:40.888008 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d659c5a4f7a446c6aaf2283f65b45106 because the tablet was not found. No further retry: Not found: Tablet not found: d659c5a4f7a446c6aaf2283f65b45106
W20250112 02:52:40.888207 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 because the tablet was not found. No further retry: Not found: Tablet not found: 6913b0ac9ceb4dbc9cdf591f159134f7
W20250112 02:52:40.888412 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b6d84027ad6d40a5be5afcdd854431a5 because the tablet was not found. No further retry: Not found: Tablet not found: b6d84027ad6d40a5be5afcdd854431a5
W20250112 02:52:40.888585 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet a85300a9aa1c4c50a3957a004bea7933 because the tablet was not found. No further retry: Not found: Tablet not found: a85300a9aa1c4c50a3957a004bea7933
W20250112 02:52:40.888814 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 352c87f1dc724e7b94bc2e64abfce8d7 because the tablet was not found. No further retry: Not found: Tablet not found: 352c87f1dc724e7b94bc2e64abfce8d7
W20250112 02:52:40.889001 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 1b184806b4884cd3b0a002905a1e57af because the tablet was not found. No further retry: Not found: Tablet not found: 1b184806b4884cd3b0a002905a1e57af
W20250112 02:52:40.889210 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 1dda52f012794493a102ed1abc718f5b because the tablet was not found. No further retry: Not found: Tablet not found: 1dda52f012794493a102ed1abc718f5b
W20250112 02:52:40.889372 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 9af55dab202444358ea1a0674d37946d because the tablet was not found. No further retry: Not found: Tablet not found: 9af55dab202444358ea1a0674d37946d
W20250112 02:52:40.889552 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 73136be25c79491784d3ea71aa9b27f2 because the tablet was not found. No further retry: Not found: Tablet not found: 73136be25c79491784d3ea71aa9b27f2
W20250112 02:52:40.889708 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 6cb76746b3a846c3aba531c04bf63966 because the tablet was not found. No further retry: Not found: Tablet not found: 6cb76746b3a846c3aba531c04bf63966
W20250112 02:52:40.889875 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d4abf59d96414e5faa7cf75ba133832c because the tablet was not found. No further retry: Not found: Tablet not found: d4abf59d96414e5faa7cf75ba133832c
W20250112 02:52:40.890018 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet c8a7281c0c09442ebd0cc13b8496c86e because the tablet was not found. No further retry: Not found: Tablet not found: c8a7281c0c09442ebd0cc13b8496c86e
W20250112 02:52:40.890170 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 058a9ca089cd4967887b0a1127c7d03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 058a9ca089cd4967887b0a1127c7d03f already in progress: creating tablet
W20250112 02:52:40.890308 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d45fce86e02f46d4a57af718fa2a62c8 because the tablet was not found. No further retry: Not found: Tablet not found: d45fce86e02f46d4a57af718fa2a62c8
W20250112 02:52:40.890455 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 7a18132549374218893f35a4e4226b72 because the tablet was not found. No further retry: Not found: Tablet not found: 7a18132549374218893f35a4e4226b72
W20250112 02:52:40.890590 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b8daf4ef3593410eae87672402292be6 because the tablet was not found. No further retry: Not found: Tablet not found: b8daf4ef3593410eae87672402292be6
W20250112 02:52:40.890734 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 003a3b4705e943c6a900762ec11ddd47 because the tablet was not found. No further retry: Not found: Tablet not found: 003a3b4705e943c6a900762ec11ddd47
W20250112 02:52:40.890877 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 2730634a5cfd4c2fb5c4e30100c556d4 because the tablet was not found. No further retry: Not found: Tablet not found: 2730634a5cfd4c2fb5c4e30100c556d4
W20250112 02:52:40.891016 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b61f16f3d601436bb4b32344de28c1a7 because the tablet was not found. No further retry: Not found: Tablet not found: b61f16f3d601436bb4b32344de28c1a7
I20250112 02:52:40.894191 2208 leader_election.cc:290] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.894153 2003 ts_tablet_manager.cc:1918] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:40.894563 2003 log.cc:1198] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/3e378a6a9b4a455c9f5e1d1ec0fd6ca0
I20250112 02:52:40.895339 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a0d939fdb94590a86389f1f4693e9a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:40.895552 918 raft_consensus.cc:3054] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:40.896420 2024 ts_tablet_manager.cc:1918] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:40.896507 2024 log.cc:1198] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/8ee98f055024429eafaa29bb961cab87
I20250112 02:52:40.896425 2003 ts_tablet_manager.cc:1939] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:40.896817 2024 ts_tablet_manager.cc:1939] T 8ee98f055024429eafaa29bb961cab87 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:40.897471 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 112) took 6030 ms. Trace:
I20250112 02:52:40.897540 2024 rpcz_store.cc:276] 0112 02:52:34.866602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:34.876397 (+ 9795us) service_pool.cc:225] Handling call
0112 02:52:40.789290 (+5912893us) ts_tablet_manager.cc:1145] Deleting tablet 8ee98f055024429eafaa29bb961cab87
0112 02:52:40.789297 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.832488 (+ 43191us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.896342 (+ 63854us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.897454 (+ 1112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4783,"tablet-delete.queue_time_us":5912641}
I20250112 02:52:40.897892 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 152) took 4990 ms. Trace:
I20250112 02:52:40.897951 2024 rpcz_store.cc:276] 0112 02:52:35.907509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:35.907553 (+ 44us) service_pool.cc:225] Handling call
0112 02:52:40.897793 (+4990240us) ts_tablet_manager.cc:1145] Deleting tablet 82b351c563d749df806ab5d0331a4ef2
0112 02:52:40.897801 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.897880 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4988295}
I20250112 02:52:40.898164 2024 tablet_replica.cc:331] T 003a3b4705e943c6a900762ec11ddd47 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:40.898326 2024 raft_consensus.cc:2238] T 003a3b4705e943c6a900762ec11ddd47 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:40.898334 918 raft_consensus.cc:2463] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:40.898438 2024 raft_consensus.cc:2267] T 003a3b4705e943c6a900762ec11ddd47 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:40.898871 2024 ts_tablet_manager.cc:1905] T 003a3b4705e943c6a900762ec11ddd47 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.898887 989 leader_election.cc:304] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:40.902037 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 8ee98f055024429eafaa29bb961cab87 (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]) successfully deleted
W20250112 02:52:40.902307 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 82b351c563d749df806ab5d0331a4ef2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82b351c563d749df806ab5d0331a4ef2 already in progress: creating tablet
I20250112 02:52:40.905561 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]) successfully deleted
I20250112 02:52:40.910545 2736 raft_consensus.cc:2798] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:40.910702 2736 raft_consensus.cc:695] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:40.910872 2047 raft_consensus.cc:491] T 47a0d939fdb94590a86389f1f4693e9a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.910979 2047 raft_consensus.cc:513] T 47a0d939fdb94590a86389f1f4693e9a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:40.910967 2736 consensus_queue.cc:237] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:40.911536 2047 leader_election.cc:290] T 47a0d939fdb94590a86389f1f4693e9a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:40.912385 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a0d939fdb94590a86389f1f4693e9a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:40.912464 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a0d939fdb94590a86389f1f4693e9a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:40.912796 918 raft_consensus.cc:2388] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 289afc116aa74408aa5bff74e11c50d8 in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
I20250112 02:52:40.913272 713 leader_election.cc:304] T 47a0d939fdb94590a86389f1f4693e9a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:40.913640 2047 raft_consensus.cc:3054] T 47a0d939fdb94590a86389f1f4693e9a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:40.915644 2047 raft_consensus.cc:2743] T 47a0d939fdb94590a86389f1f4693e9a P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:40.916054 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42444e99aba84a7b873bc8cdf954a6dc" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:40.916257 775 raft_consensus.cc:2463] T 42444e99aba84a7b873bc8cdf954a6dc P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:40.916980 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42444e99aba84a7b873bc8cdf954a6dc" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:40.917153 778 raft_consensus.cc:3054] T 42444e99aba84a7b873bc8cdf954a6dc P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:40.917819 1173 tablet_bootstrap.cc:492] T bab4978edea0445fb0456e88e7fb4c44 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:40.917937 1173 ts_tablet_manager.cc:1397] T bab4978edea0445fb0456e88e7fb4c44 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.265s user 0.002s sys 0.005s
I20250112 02:52:40.918668 1173 raft_consensus.cc:357] T bab4978edea0445fb0456e88e7fb4c44 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.918839 1173 raft_consensus.cc:383] T bab4978edea0445fb0456e88e7fb4c44 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.918887 1173 raft_consensus.cc:738] T bab4978edea0445fb0456e88e7fb4c44 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.919102 1173 consensus_queue.cc:260] T bab4978edea0445fb0456e88e7fb4c44 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.919755 1173 ts_tablet_manager.cc:1428] T bab4978edea0445fb0456e88e7fb4c44 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:40.920074 1173 tablet_bootstrap.cc:492] T c73f2fa452b84be398476b8aba6c9a08 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:40.920818 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a0d939fdb94590a86389f1f4693e9a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:40.921141 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 3b9280db29e54d41af1bed2fa5aa5535
I20250112 02:52:40.921538 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a0d939fdb94590a86389f1f4693e9a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:40.921694 455 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40610 (request call id 162) took 2939 ms. Trace:
I20250112 02:52:40.921810 455 rpcz_store.cc:276] 0112 02:52:37.982664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.367568 (+2384904us) service_pool.cc:225] Handling call
0112 02:52:40.368027 (+ 459us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:40.368163 (+ 136us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:52:40.368216 (+ 53us) catalog_manager.cc:2762] Locking tablets
0112 02:52:40.368348 (+ 132us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:52:40.463542 (+ 95194us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:52:40.463609 (+ 67us) catalog_manager.cc:2810] Committing in-memory state
0112 02:52:40.921092 (+457483us) catalog_manager.cc:2814] Aborting table tasks
0112 02:52:40.921638 (+ 546us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:40.369429 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.369511 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.369516 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:40.369519 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:40.369602 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.369610 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.369614 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:40.369694 (+ 80us) tablet.cc:698] Row locks acquired
0112 02:52:40.369696 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:40.369720 (+ 24us) write_op.cc:270] Start()
0112 02:52:40.369741 (+ 21us) write_op.cc:276] Timestamp: P: 1736650360369663 usec, L: 0
0112 02:52:40.369744 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.369864 (+ 120us) log.cc:844] Serialized 725 byte log entry
0112 02:52:40.456427 (+ 86563us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.457127 (+ 700us) write_op.cc:301] APPLY: starting
0112 02:52:40.457136 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.457166 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.457193 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.457246 (+ 53us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.457279 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:40.457284 (+ 5us) write_op.cc:312] APPLY: finished
0112 02:52:40.457327 (+ 43us) log.cc:844] Serialized 29 byte log entry
0112 02:52:40.457358 (+ 31us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.457369 (+ 11us) write_op.cc:454] Released schema lock
0112 02:52:40.457373 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:40.457423 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.457426 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":668,"apply.run_cpu_time_us":292,"apply.run_wall_time_us":292,"num_ops":2,"prepare.queue_time_us":567,"prepare.run_cpu_time_us":475,"prepare.run_wall_time_us":474,"replication_time_us":86665}]]}
I20250112 02:52:40.922485 778 raft_consensus.cc:2463] T 42444e99aba84a7b873bc8cdf954a6dc P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:40.922622 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_4657847b84384e628194a8ff43e183a1" } new_table_name: "default.Table_1325d9a6747941458d8655d565170565" modify_external_catalogs: true
I20250112 02:52:40.923534 1177 tablet_bootstrap.cc:492] T 8c821ea3ca4946c290c17135366e0cac P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:40.923729 1177 ts_tablet_manager.cc:1397] T 8c821ea3ca4946c290c17135366e0cac P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.154s user 0.002s sys 0.001s
I20250112 02:52:40.924787 772 raft_consensus.cc:2463] T 47a0d939fdb94590a86389f1f4693e9a P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:40.924804 1177 raft_consensus.cc:357] T 8c821ea3ca4946c290c17135366e0cac P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.924979 1177 raft_consensus.cc:383] T 8c821ea3ca4946c290c17135366e0cac P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.925026 1177 raft_consensus.cc:738] T 8c821ea3ca4946c290c17135366e0cac P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.925226 2047 raft_consensus.cc:491] T 46d1beac4410422bbf1e912d35999f3d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:40.925323 2047 raft_consensus.cc:513] T 46d1beac4410422bbf1e912d35999f3d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.925899 2047 leader_election.cc:290] T 46d1beac4410422bbf1e912d35999f3d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:40.926713 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d1beac4410422bbf1e912d35999f3d" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:40.927803 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d1beac4410422bbf1e912d35999f3d" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:40.928265 713 leader_election.cc:304] T 46d1beac4410422bbf1e912d35999f3d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:40.928666 2047 raft_consensus.cc:2743] T 46d1beac4410422bbf1e912d35999f3d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:40.929301 1177 consensus_queue.cc:260] T 8c821ea3ca4946c290c17135366e0cac P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.931885 1177 ts_tablet_manager.cc:1428] T 8c821ea3ca4946c290c17135366e0cac P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250112 02:52:40.932217 1177 tablet_bootstrap.cc:492] T 777272e7159b438ebff4db93698411d0 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:40.936237 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b9280db29e54d41af1bed2fa5aa5535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:40.936587 2208 consensus_queue.cc:1035] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:40.937685 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b9280db29e54d41af1bed2fa5aa5535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:40.933671 1177 tablet_bootstrap.cc:654] T 777272e7159b438ebff4db93698411d0 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.938122 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b9280db29e54d41af1bed2fa5aa5535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:40.956549 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 3b9280db29e54d41af1bed2fa5aa5535 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b9280db29e54d41af1bed2fa5aa5535 already in progress: creating tablet
I20250112 02:52:40.958223 2024 ts_tablet_manager.cc:1918] T 003a3b4705e943c6a900762ec11ddd47 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:40.958333 2024 log.cc:1198] T 003a3b4705e943c6a900762ec11ddd47 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/003a3b4705e943c6a900762ec11ddd47
I20250112 02:52:40.958611 2024 ts_tablet_manager.cc:1939] T 003a3b4705e943c6a900762ec11ddd47 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:40.958892 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 154) took 5043 ms. Trace:
I20250112 02:52:40.958978 2024 rpcz_store.cc:276] 0112 02:52:35.915826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:35.915866 (+ 40us) service_pool.cc:225] Handling call
0112 02:52:40.898130 (+4982264us) ts_tablet_manager.cc:1145] Deleting tablet 003a3b4705e943c6a900762ec11ddd47
0112 02:52:40.898136 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.906582 (+ 8446us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.958125 (+ 51543us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.958878 (+ 753us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4965065}
I20250112 02:52:40.959414 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 003a3b4705e943c6a900762ec11ddd47 (table default.Table_743b3f7029534564b35c0c137b35fd75 [id=b2fa3922f6d447769a8179052f9a1861]) successfully deleted
I20250112 02:52:40.965319 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 171) took 3184 ms. Trace:
I20250112 02:52:40.965436 2024 rpcz_store.cc:276] 0112 02:52:37.780400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:37.781634 (+ 1234us) service_pool.cc:225] Handling call
0112 02:52:40.965191 (+3183557us) ts_tablet_manager.cc:1145] Deleting tablet 13578542e9e741e8817e0ee63b10f1a1
0112 02:52:40.965200 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.965301 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3183362}
W20250112 02:52:40.965910 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 13578542e9e741e8817e0ee63b10f1a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13578542e9e741e8817e0ee63b10f1a1 already in progress: creating tablet
I20250112 02:52:40.966100 2562 ts_tablet_manager.cc:1918] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 02:52:40.966292 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 172) took 2954 ms. Trace:
I20250112 02:52:40.966368 2024 rpcz_store.cc:276] 0112 02:52:38.012025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.012079 (+ 54us) service_pool.cc:225] Handling call
0112 02:52:40.966180 (+2954101us) ts_tablet_manager.cc:1145] Deleting tablet a3ffdc237f8640f0b378809f93fc6292
0112 02:52:40.966189 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.966263 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2953924}
W20250112 02:52:40.966742 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet a3ffdc237f8640f0b378809f93fc6292 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3ffdc237f8640f0b378809f93fc6292 already in progress: creating tablet
I20250112 02:52:40.971884 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 173) took 2703 ms. Trace:
I20250112 02:52:40.972023 2024 rpcz_store.cc:276] 0112 02:52:38.267932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.267989 (+ 57us) service_pool.cc:225] Handling call
0112 02:52:40.971790 (+2703801us) ts_tablet_manager.cc:1145] Deleting tablet cbe30f9860e2492b90e8cedf8ff9daba
0112 02:52:40.971799 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.971869 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2703640}
W20250112 02:52:40.972414 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet cbe30f9860e2492b90e8cedf8ff9daba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbe30f9860e2492b90e8cedf8ff9daba already in progress: creating tablet
I20250112 02:52:40.972751 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 183) took 2110 ms. Trace:
I20250112 02:52:40.972862 2024 rpcz_store.cc:276] 0112 02:52:38.861966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.862018 (+ 52us) service_pool.cc:225] Handling call
0112 02:52:40.972653 (+2110635us) ts_tablet_manager.cc:1145] Deleting tablet bd5b3c712c0f448ea9b14ab93dd731ab
0112 02:52:40.972661 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.972735 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2110427}
W20250112 02:52:40.973274 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet bd5b3c712c0f448ea9b14ab93dd731ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd5b3c712c0f448ea9b14ab93dd731ab already in progress: creating tablet
I20250112 02:52:40.973609 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 184) took 2111 ms. Trace:
I20250112 02:52:40.973698 2024 rpcz_store.cc:276] 0112 02:52:38.862258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.862320 (+ 62us) service_pool.cc:225] Handling call
0112 02:52:40.973509 (+2111189us) ts_tablet_manager.cc:1145] Deleting tablet b1d9fae290ec4281839991f9d40b105c
0112 02:52:40.973517 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.973583 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2111013}
W20250112 02:52:40.974090 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b1d9fae290ec4281839991f9d40b105c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1d9fae290ec4281839991f9d40b105c already in progress: creating tablet
I20250112 02:52:40.974347 2024 tablet_replica.cc:331] T bab4978edea0445fb0456e88e7fb4c44 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:40.976400 2024 raft_consensus.cc:2238] T bab4978edea0445fb0456e88e7fb4c44 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:40.976568 2024 raft_consensus.cc:2267] T bab4978edea0445fb0456e88e7fb4c44 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:40.977288 2024 ts_tablet_manager.cc:1905] T bab4978edea0445fb0456e88e7fb4c44 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:40.978819 1177 tablet_bootstrap.cc:492] T 777272e7159b438ebff4db93698411d0 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:40.981058 1173 tablet_bootstrap.cc:654] T c73f2fa452b84be398476b8aba6c9a08 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:40.991127 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:40.995302 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:40.995826 449 catalog_manager.cc:5526] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:40.997558 1177 ts_tablet_manager.cc:1397] T 777272e7159b438ebff4db93698411d0 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.001s
I20250112 02:52:40.997573 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:40.998148 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:40.998426 1177 raft_consensus.cc:357] T 777272e7159b438ebff4db93698411d0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:40.998661 1177 raft_consensus.cc:383] T 777272e7159b438ebff4db93698411d0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:40.998725 1177 raft_consensus.cc:738] T 777272e7159b438ebff4db93698411d0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:40.999862 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:41.000368 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet ead501d76bb1412e9839fb36b9911e6f (table default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3 [id=c1fbaa09536c4fa994c6949a3fd58c75]). Expected version 1 got 0
I20250112 02:52:41.001430 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_6e30b071de75435da0245b469fb16ff1 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 1 got 0
I20250112 02:52:41.002249 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet ca2165362ee541adac724b63f39b1489 (table default.Table_41b7056b6e9240928d0af52aec2e9338 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 2 got 0
I20250112 02:52:41.003237 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:41.004194 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet d95f98ae7bdf42518bbff55e257fea88 (table default.Table_2fcdbb05970f4a098f7ba3e287fa0348 [id=70c61753e9944e1fbd65e9ec41a48ee2]). Expected version 1 got 0
I20250112 02:52:41.005399 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:41.005491 2562 log.cc:1198] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/f31b777e7e0f43a28839d8c48c6a6590
I20250112 02:52:41.006038 2562 ts_tablet_manager.cc:1939] T f31b777e7e0f43a28839d8c48c6a6590 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:41.005242 1181 tablet_bootstrap.cc:492] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:41.006266 1181 ts_tablet_manager.cc:1397] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.368s user 0.000s sys 0.006s
I20250112 02:52:41.008100 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:41.008141 1181 raft_consensus.cc:357] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.008359 1181 raft_consensus.cc:383] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.008426 2696 raft_consensus.cc:491] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.008584 2696 raft_consensus.cc:513] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.009219 2696 leader_election.cc:290] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:41.009748 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164f76edcf8f4bd498dcd2c25fd7477d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:41.010308 855 leader_election.cc:343] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.008430 1181 raft_consensus.cc:738] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
W20250112 02:52:41.010517 854 leader_election.cc:343] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 164f76edcf8f4bd498dcd2c25fd7477d
I20250112 02:52:41.010605 854 leader_election.cc:304] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:41.010689 1181 consensus_queue.cc:260] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.010918 2673 raft_consensus.cc:2743] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:41.011427 1181 ts_tablet_manager.cc:1428] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 02:52:41.011770 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:41.012655 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:41.012766 1181 tablet_bootstrap.cc:492] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:41.013258 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:41.015249 1181 tablet_bootstrap.cc:654] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.016475 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 235) took 1672 ms. Trace:
I20250112 02:52:41.016592 2562 rpcz_store.cc:276] 0112 02:52:39.343710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.343769 (+ 59us) service_pool.cc:225] Handling call
0112 02:52:40.683916 (+1340147us) ts_tablet_manager.cc:1145] Deleting tablet f31b777e7e0f43a28839d8c48c6a6590
0112 02:52:40.683924 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.691001 (+ 7077us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.965988 (+274987us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.016459 (+ 50471us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1339957}
I20250112 02:52:41.017181 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet f31b777e7e0f43a28839d8c48c6a6590 (table default.Table_41b7056b6e9240928d0af52aec2e9338 [id=6402abce8d1d45c182cc989490c6a5f0]) successfully deleted
I20250112 02:52:41.017974 2562 tablet_replica.cc:331] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:41.018146 449 catalog_manager.cc:5526] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
W20250112 02:52:41.019125 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet f3771166b77e4f90aaa116ab1a9dc299 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3771166b77e4f90aaa116ab1a9dc299 already in progress: creating tablet
I20250112 02:52:41.019179 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:41.019426 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:41.020447 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:41.021495 449 catalog_manager.cc:5526] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:41.022928 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:41.023648 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:41.027298 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:41.028002 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 17c54c5dc86446cf85974777c8145ef6 (table default.Table_8ab566cf9c8f45eba4dc9bb85264c823 [id=a622f7a973344a388b3275adbcce6234]). Expected version 1 got 0
I20250112 02:52:40.988241 2024 ts_tablet_manager.cc:1918] T bab4978edea0445fb0456e88e7fb4c44 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:41.028956 449 catalog_manager.cc:5526] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:41.029206 2024 log.cc:1198] T bab4978edea0445fb0456e88e7fb4c44 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/bab4978edea0445fb0456e88e7fb4c44
I20250112 02:52:41.029310 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:41.029569 2024 ts_tablet_manager.cc:1939] T bab4978edea0445fb0456e88e7fb4c44 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:41.029654 449 catalog_manager.cc:5526] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:41.031855 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
I20250112 02:52:41.032569 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:41.033115 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:41.034847 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_328664e3c01f483ca8e5ff50a497f68e" } modify_external_catalogs: true
I20250112 02:52:41.035663 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_328664e3c01f483ca8e5ff50a497f68e" } modify_external_catalogs: true
I20250112 02:52:41.038708 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_365739770a11434dbf260ea912e7ea81 [id=868adb627654450c936ef198e49d853b] alter complete (version 1)
I20250112 02:52:41.038820 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet ff1b3ee229c445b19405cf4a19cf8003
I20250112 02:52:41.039326 447 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40610 (request call id 164) took 2570 ms. Trace:
I20250112 02:52:41.039445 447 rpcz_store.cc:276] 0112 02:52:38.468566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.376641 (+1908075us) service_pool.cc:225] Handling call
0112 02:52:40.389357 (+ 12716us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd73cf6 00007f3e13e76ef6 00007f3e13e76c25 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:40.389640 (+ 283us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:40.389743 (+ 103us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:52:40.389787 (+ 44us) catalog_manager.cc:2762] Locking tablets
0112 02:52:40.389900 (+ 113us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:52:40.475236 (+ 85336us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:52:40.475283 (+ 47us) catalog_manager.cc:2810] Committing in-memory state
0112 02:52:41.038739 (+563456us) catalog_manager.cc:2814] Aborting table tasks
0112 02:52:41.039289 (+ 550us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:40.390473 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.390618 (+ 145us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.390625 (+ 7us) write_op.cc:435] Acquired schema lock
0112 02:52:40.390629 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:40.390725 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.390732 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.390736 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:40.390819 (+ 83us) tablet.cc:698] Row locks acquired
0112 02:52:40.390823 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:40.390848 (+ 25us) write_op.cc:270] Start()
0112 02:52:40.390870 (+ 22us) write_op.cc:276] Timestamp: P: 1736650360390788 usec, L: 0
0112 02:52:40.390873 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.390995 (+ 122us) log.cc:844] Serialized 725 byte log entry
0112 02:52:40.473510 (+ 82515us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.474436 (+ 926us) write_op.cc:301] APPLY: starting
0112 02:52:40.474446 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.474471 (+ 25us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.474500 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.474551 (+ 51us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.474583 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:40.474592 (+ 9us) write_op.cc:312] APPLY: finished
0112 02:52:40.474642 (+ 50us) log.cc:844] Serialized 29 byte log entry
0112 02:52:40.474672 (+ 30us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.474683 (+ 11us) write_op.cc:454] Released schema lock
0112 02:52:40.474688 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":28789120,"child_traces":[["op",{"apply.queue_time_us":898,"apply.run_cpu_time_us":294,"apply.run_wall_time_us":294,"num_ops":2,"prepare.queue_time_us":79,"prepare.run_cpu_time_us":517,"prepare.run_wall_time_us":567,"replication_time_us":82619,"thread_start_us":37130,"threads_started":1}]]}
I20250112 02:52:41.040220 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_65ec5f3d51f5442c91c4ba7c714cd0e3" } new_table_name: "default.Table_2add53a7889440009ab2d3b0e58c5381" modify_external_catalogs: true
I20250112 02:52:41.043628 1177 consensus_queue.cc:260] T 777272e7159b438ebff4db93698411d0 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.044193 1177 ts_tablet_manager.cc:1428] T 777272e7159b438ebff4db93698411d0 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.046s user 0.001s sys 0.001s
I20250112 02:52:41.046434 743 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1b3ee229c445b19405cf4a19cf8003 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.047027 1177 tablet_bootstrap.cc:492] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:41.047299 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet ff1b3ee229c445b19405cf4a19cf8003 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff1b3ee229c445b19405cf4a19cf8003 already in progress: creating tablet
I20250112 02:52:41.047683 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 185) took 2185 ms. Trace:
I20250112 02:52:41.047813 2024 rpcz_store.cc:276] 0112 02:52:38.862512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.862559 (+ 47us) service_pool.cc:225] Handling call
0112 02:52:40.974310 (+2111751us) ts_tablet_manager.cc:1145] Deleting tablet bab4978edea0445fb0456e88e7fb4c44
0112 02:52:40.974318 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:40.980237 (+ 5919us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:40.988164 (+ 7927us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.047662 (+ 59498us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2111552}
I20250112 02:52:41.048137 2024 tablet_replica.cc:331] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:41.048305 2024 raft_consensus.cc:2238] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:41.048436 2024 raft_consensus.cc:2267] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:41.048954 2024 ts_tablet_manager.cc:1905] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:41.049322 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet bab4978edea0445fb0456e88e7fb4c44 (table default.Table_9dd5993a01b44a0abe33a5738c8d5c2b [id=6a7dcf7429b24699b036c53b3005051f]) successfully deleted
I20250112 02:52:41.049808 2562 raft_consensus.cc:2238] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:41.050030 2562 raft_consensus.cc:2267] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:41.050865 2562 ts_tablet_manager.cc:1905] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:41.053418 895 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1b3ee229c445b19405cf4a19cf8003 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:41.053664 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1b3ee229c445b19405cf4a19cf8003 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:41.054278 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 118976us
I20250112 02:52:41.054700 2024 ts_tablet_manager.cc:1918] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:41.054807 2024 log.cc:1198] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/8c821ea3ca4946c290c17135366e0cac
I20250112 02:52:41.055181 2024 ts_tablet_manager.cc:1939] T 8c821ea3ca4946c290c17135366e0cac P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:41.062500 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_6e30b071de75435da0245b469fb16ff1" } new_table_name: "default.Table_b9dff6cb7bbd4d0a88de458399b0d2c0" modify_external_catalogs: true
I20250112 02:52:41.063800 1173 tablet_bootstrap.cc:492] T c73f2fa452b84be398476b8aba6c9a08 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:41.063930 1173 ts_tablet_manager.cc:1397] T c73f2fa452b84be398476b8aba6c9a08 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.144s user 0.003s sys 0.000s
I20250112 02:52:41.056540 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 186) took 2193 ms. Trace:
I20250112 02:52:41.064749 1173 raft_consensus.cc:357] T c73f2fa452b84be398476b8aba6c9a08 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.064944 1173 raft_consensus.cc:383] T c73f2fa452b84be398476b8aba6c9a08 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.064821 2024 rpcz_store.cc:276] 0112 02:52:38.862923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.862966 (+ 43us) service_pool.cc:225] Handling call
0112 02:52:41.048095 (+2185129us) ts_tablet_manager.cc:1145] Deleting tablet 8c821ea3ca4946c290c17135366e0cac
0112 02:52:41.048104 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.051865 (+ 3761us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.054626 (+ 2761us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.056522 (+ 1896us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2184984}
I20250112 02:52:41.065001 1173 raft_consensus.cc:738] T c73f2fa452b84be398476b8aba6c9a08 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.065235 1173 consensus_queue.cc:260] T c73f2fa452b84be398476b8aba6c9a08 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.065423 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 8c821ea3ca4946c290c17135366e0cac (table default.Table_0f1762619e9543018fd2df57adb6367e [id=5f5a61b2a00d4d29bd79a696474a787b]) successfully deleted
I20250112 02:52:41.066109 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 187) took 2202 ms. Trace:
I20250112 02:52:41.066175 2024 rpcz_store.cc:276] 0112 02:52:38.863256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.863298 (+ 42us) service_pool.cc:225] Handling call
0112 02:52:41.065978 (+2202680us) ts_tablet_manager.cc:1145] Deleting tablet f776e2a6ec704cf9b64483bc7d5b1dff
0112 02:52:41.065988 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.066092 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2202514}
W20250112 02:52:41.066576 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f776e2a6ec704cf9b64483bc7d5b1dff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f776e2a6ec704cf9b64483bc7d5b1dff already in progress: creating tablet
I20250112 02:52:41.071491 1181 tablet_bootstrap.cc:492] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:41.072893 1173 ts_tablet_manager.cc:1428] T c73f2fa452b84be398476b8aba6c9a08 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250112 02:52:41.073783 1173 tablet_bootstrap.cc:492] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:41.073969 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_15bcb2c0f2a74458a1bed231fd3437fc [id=ca9c8281bf254f248076cb600e7c4f87] alter complete (version 2)
I20250112 02:52:41.073182 1181 ts_tablet_manager.cc:1397] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.060s user 0.002s sys 0.003s
I20250112 02:52:41.077203 2024 tablet_replica.cc:331] T 06edfca60eed4dd3a2dee7b662b0970c P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:41.080997 1181 raft_consensus.cc:357] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.081748 1181 raft_consensus.cc:383] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.082119 1181 raft_consensus.cc:738] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
W20250112 02:52:41.082413 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99407ead9cf547e3bbe78c0d0b327e6a consensus state has no leader
I20250112 02:52:41.082888 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_ea881fde8d694f71b5faf60948630d96" } modify_external_catalogs: true
I20250112 02:52:41.083072 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_ea881fde8d694f71b5faf60948630d96" } modify_external_catalogs: true
I20250112 02:52:41.082816 1181 consensus_queue.cc:260] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.088243 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet d0cb54003a83472cb9c57ace3e65b0be
I20250112 02:52:41.088846 454 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40610 (request call id 163) took 2876 ms. Trace:
I20250112 02:52:41.088943 454 rpcz_store.cc:276] 0112 02:52:38.212769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.375658 (+2162889us) service_pool.cc:225] Handling call
0112 02:52:40.382123 (+ 6465us) spinlock_profiling.cc:243] Waited 6.17 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd73cf6 00007f3e13e76ef6 00007f3e13e76c25 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:40.382580 (+ 457us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:40.382742 (+ 162us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:52:40.382802 (+ 60us) catalog_manager.cc:2762] Locking tablets
0112 02:52:40.382995 (+ 193us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:52:40.475054 (+ 92059us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:52:40.475138 (+ 84us) catalog_manager.cc:2810] Committing in-memory state
0112 02:52:41.088194 (+613056us) catalog_manager.cc:2814] Aborting table tasks
0112 02:52:41.088794 (+ 600us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:40.386481 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.386561 (+ 80us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.386565 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:40.386569 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:40.386671 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.386678 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.386682 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:40.386795 (+ 113us) tablet.cc:698] Row locks acquired
0112 02:52:40.386798 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:40.386826 (+ 28us) write_op.cc:270] Start()
0112 02:52:40.386849 (+ 23us) write_op.cc:276] Timestamp: P: 1736650360386766 usec, L: 0
0112 02:52:40.386852 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.386992 (+ 140us) log.cc:844] Serialized 725 byte log entry
0112 02:52:40.473383 (+ 86391us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.473670 (+ 287us) write_op.cc:301] APPLY: starting
0112 02:52:40.473683 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.473733 (+ 50us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.473763 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.473816 (+ 53us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.473843 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:40.473849 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:40.473907 (+ 58us) log.cc:844] Serialized 29 byte log entry
0112 02:52:40.473957 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.473971 (+ 14us) write_op.cc:454] Released schema lock
0112 02:52:40.473976 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:40.474043 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.474047 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":14188416,"child_traces":[["op",{"apply.queue_time_us":240,"apply.run_cpu_time_us":367,"apply.run_wall_time_us":366,"num_ops":2,"prepare.queue_time_us":2094,"prepare.run_cpu_time_us":578,"prepare.run_wall_time_us":577,"replication_time_us":86504}]]}
I20250112 02:52:41.089716 1177 tablet_bootstrap.cc:654] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.089808 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_ed3c26ab0711404581dcd62adb343dde" } new_table_name: "default.Table_66ecd84e49d64f50b63b520f74b8e062" modify_external_catalogs: true
I20250112 02:52:41.095489 1181 ts_tablet_manager.cc:1428] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20250112 02:52:41.097025 743 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cb54003a83472cb9c57ace3e65b0be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:41.098182 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d0cb54003a83472cb9c57ace3e65b0be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0cb54003a83472cb9c57ace3e65b0be already in progress: creating tablet
I20250112 02:52:41.098199 1181 tablet_bootstrap.cc:492] T 9bbef42e11a843b18ed582ab4d04a92a P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:41.099359 895 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cb54003a83472cb9c57ace3e65b0be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:38786
W20250112 02:52:41.106302 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46d1beac4410422bbf1e912d35999f3d consensus state has no leader
I20250112 02:52:41.106691 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_3185108f36ed429e9a9967cc2b11072e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:41.107663 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3185108f36ed429e9a9967cc2b11072e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:41.113476 1181 tablet_bootstrap.cc:654] T 9bbef42e11a843b18ed582ab4d04a92a P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.121109 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cb54003a83472cb9c57ace3e65b0be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.123656 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 681a930541a84f0197ecafe670278e70 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:41.126902 895 tablet_service.cc:1514] Processing DeleteTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:41.127190 2208 raft_consensus.cc:491] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.127295 2208 raft_consensus.cc:513] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.127866 2208 leader_election.cc:290] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:41.128119 2024 raft_consensus.cc:2238] T 06edfca60eed4dd3a2dee7b662b0970c P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:41.128291 2024 raft_consensus.cc:2267] T 06edfca60eed4dd3a2dee7b662b0970c P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:41.128345 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf54f5185a0f4c70b9057309fca14e56" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:41.128510 910 raft_consensus.cc:2463] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:41.128840 2024 ts_tablet_manager.cc:1905] T 06edfca60eed4dd3a2dee7b662b0970c P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:41.128887 989 leader_election.cc:304] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:41.129276 2326 raft_consensus.cc:2798] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:41.129338 2326 raft_consensus.cc:491] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:41.129390 2326 raft_consensus.cc:3054] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.129540 1173 tablet_bootstrap.cc:654] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.131078 2755 tablet.cc:1721] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:41.131886 2326 raft_consensus.cc:513] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.132407 2326 leader_election.cc:290] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:41.133247 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf54f5185a0f4c70b9057309fca14e56" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:41.133386 910 raft_consensus.cc:3054] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.134634 2756 tablet.cc:1721] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:41.135510 2024 ts_tablet_manager.cc:1918] T 06edfca60eed4dd3a2dee7b662b0970c P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:41.136227 910 raft_consensus.cc:2463] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:41.137924 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf54f5185a0f4c70b9057309fca14e56" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:41.136698 989 leader_election.cc:304] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
W20250112 02:52:41.138491 989 consensus_peers.cc:487] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f31b777e7e0f43a28839d8c48c6a6590. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:41.138644 2208 raft_consensus.cc:2798] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:41.138769 2208 raft_consensus.cc:695] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:41.139012 2208 consensus_queue.cc:237] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.139382 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a34d03a80ae402db6607261a5c40b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:41.140241 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf54f5185a0f4c70b9057309fca14e56" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:41.140344 2024 log.cc:1198] T 06edfca60eed4dd3a2dee7b662b0970c P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/06edfca60eed4dd3a2dee7b662b0970c
I20250112 02:52:41.140647 2024 ts_tablet_manager.cc:1939] T 06edfca60eed4dd3a2dee7b662b0970c P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:41.141144 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e7882e07784bdda18dc1ffd20963c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:41.141438 991 leader_election.cc:343] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.143060 895 tablet_service.cc:1514] Processing DeleteTablet for tablet f31b777e7e0f43a28839d8c48c6a6590 with delete_type TABLET_DATA_DELETED (replaced by tablet ca2165362ee541adac724b63f39b1489) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:41.142062 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5b3c712c0f448ea9b14ab93dd731ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:41.143596 991 consensus_peers.cc:487] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4348fe4c32704bf7b3a8d21ff3b8ac1b. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:41.145133 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet c29a9ca43e034b43a00207e5847bbee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.145651 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet b1d9fae290ec4281839991f9d40b105c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:41.145994 991 leader_election.cc:343] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.147189 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af55dab202444358ea1a0674d37946d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:41.146478 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet bab4978edea0445fb0456e88e7fb4c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.147905 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c821ea3ca4946c290c17135366e0cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.148867 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 058a9ca089cd4967887b0a1127c7d03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:41.149538 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.150962 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 188) took 2287 ms. Trace:
I20250112 02:52:41.151031 2762 tablet.cc:1721] T ead501d76bb1412e9839fb36b9911e6f P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:41.151095 2024 rpcz_store.cc:276] 0112 02:52:38.863587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.863648 (+ 61us) service_pool.cc:225] Handling call
0112 02:52:41.077164 (+2213516us) ts_tablet_manager.cc:1145] Deleting tablet 06edfca60eed4dd3a2dee7b662b0970c
0112 02:52:41.077172 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.132285 (+ 55113us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.135421 (+ 3136us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.150949 (+ 15528us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50733,"tablet-delete.queue_time_us":2213355}
I20250112 02:52:41.151751 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 06edfca60eed4dd3a2dee7b662b0970c (table default.Table_2a2995d7964346c08fcbff96b4cc3ca5 [id=70371e156f2f401ab5ee0e4e4c7545b9]) successfully deleted
I20250112 02:52:41.152169 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 189) took 2288 ms. Trace:
I20250112 02:52:41.152289 2024 rpcz_store.cc:276] 0112 02:52:38.863840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.863888 (+ 48us) service_pool.cc:225] Handling call
0112 02:52:41.152071 (+2288183us) ts_tablet_manager.cc:1145] Deleting tablet f234256b343f447dadfc3d1936a824ee
0112 02:52:41.152084 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.152153 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2288000}
I20250112 02:52:41.153435 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 190) took 2289 ms. Trace:
W20250112 02:52:41.153792 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f234256b343f447dadfc3d1936a824ee because the tablet was not found. No further retry: Not found: Tablet not found: f234256b343f447dadfc3d1936a824ee
I20250112 02:52:41.153724 2024 rpcz_store.cc:276] 0112 02:52:38.864174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.864229 (+ 55us) service_pool.cc:225] Handling call
0112 02:52:41.153369 (+2289140us) ts_tablet_manager.cc:1145] Deleting tablet b61f16f3d601436bb4b32344de28c1a7
0112 02:52:41.153377 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.153424 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2289006}
I20250112 02:52:41.153875 895 tablet_service.cc:1514] Processing DeleteTablet for tablet f3771166b77e4f90aaa116ab1a9dc299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:39 UTC) from {username='slave'} at 127.0.0.1:38786
W20250112 02:52:41.154093 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b61f16f3d601436bb4b32344de28c1a7 because the tablet was not found. No further retry: Not found: Tablet not found: b61f16f3d601436bb4b32344de28c1a7
I20250112 02:52:41.154950 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f4bd2df0ce45dabc55373f1154d3ac with delete_type TABLET_DATA_DELETED (replaced by tablet f559dedfb7854cf890bef362c50b4213) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.155293 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b351c563d749df806ab5d0331a4ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.155452 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet f234256b343f447dadfc3d1936a824ee with delete_type TABLET_DATA_DELETED (replaced by tablet 82b351c563d749df806ab5d0331a4ef2) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.155717 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 191) took 2291 ms. Trace:
I20250112 02:52:41.155791 2024 rpcz_store.cc:276] 0112 02:52:38.864366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.864406 (+ 40us) service_pool.cc:225] Handling call
0112 02:52:41.155649 (+2291243us) ts_tablet_manager.cc:1145] Deleting tablet a12c5d58e6c246c983e8a533906ce77d
0112 02:52:41.155657 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.155703 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2291095}
I20250112 02:52:41.156148 1177 tablet_bootstrap.cc:492] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:41.156412 1177 ts_tablet_manager.cc:1397] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.109s user 0.002s sys 0.007s
I20250112 02:52:41.155964 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 06edfca60eed4dd3a2dee7b662b0970c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:41.159950 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet a12c5d58e6c246c983e8a533906ce77d because the tablet was not found. No further retry: Not found: Tablet not found: a12c5d58e6c246c983e8a533906ce77d
I20250112 02:52:41.160895 1177 raft_consensus.cc:357] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.161260 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 192) took 2296 ms. Trace:
I20250112 02:52:41.161546 2024 rpcz_store.cc:276] 0112 02:52:38.864788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.864834 (+ 46us) service_pool.cc:225] Handling call
0112 02:52:41.161188 (+2296354us) ts_tablet_manager.cc:1145] Deleting tablet 2730634a5cfd4c2fb5c4e30100c556d4
0112 02:52:41.161195 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.161244 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2296188}
W20250112 02:52:41.162065 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 2730634a5cfd4c2fb5c4e30100c556d4 because the tablet was not found. No further retry: Not found: Tablet not found: 2730634a5cfd4c2fb5c4e30100c556d4
I20250112 02:52:41.162415 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 193) took 2297 ms. Trace:
I20250112 02:52:41.162711 2024 rpcz_store.cc:276] 0112 02:52:38.865172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.865211 (+ 39us) service_pool.cc:225] Handling call
0112 02:52:41.162344 (+2297133us) ts_tablet_manager.cc:1145] Deleting tablet d45fce86e02f46d4a57af718fa2a62c8
0112 02:52:41.162352 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.162400 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2296973}
W20250112 02:52:41.163266 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet d45fce86e02f46d4a57af718fa2a62c8 because the tablet was not found. No further retry: Not found: Tablet not found: d45fce86e02f46d4a57af718fa2a62c8
I20250112 02:52:41.163762 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 195) took 2280 ms. Trace:
I20250112 02:52:41.163966 1177 raft_consensus.cc:383] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.164073 2024 rpcz_store.cc:276] 0112 02:52:38.883215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.883260 (+ 45us) service_pool.cc:225] Handling call
0112 02:52:41.163578 (+2280318us) ts_tablet_manager.cc:1145] Deleting tablet 6cb76746b3a846c3aba531c04bf63966
0112 02:52:41.163704 (+ 126us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.163750 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2280119}
I20250112 02:52:41.164263 1177 raft_consensus.cc:738] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
W20250112 02:52:41.164691 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 6cb76746b3a846c3aba531c04bf63966 because the tablet was not found. No further retry: Not found: Tablet not found: 6cb76746b3a846c3aba531c04bf63966
I20250112 02:52:41.165043 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 196) took 2280 ms. Trace:
I20250112 02:52:41.165315 1177 consensus_queue.cc:260] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.165673 2024 rpcz_store.cc:276] 0112 02:52:38.884790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.884835 (+ 45us) service_pool.cc:225] Handling call
0112 02:52:41.164980 (+2280145us) ts_tablet_manager.cc:1145] Deleting tablet 73136be25c79491784d3ea71aa9b27f2
0112 02:52:41.164988 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.165030 (+ 42us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2279958}
I20250112 02:52:41.166330 1177 ts_tablet_manager.cc:1428] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
W20250112 02:52:41.166662 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 73136be25c79491784d3ea71aa9b27f2 because the tablet was not found. No further retry: Not found: Tablet not found: 73136be25c79491784d3ea71aa9b27f2
I20250112 02:52:41.167013 2024 tablet_replica.cc:331] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:41.167250 1173 tablet_bootstrap.cc:492] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:41.167373 2024 raft_consensus.cc:2238] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:41.167742 2024 raft_consensus.cc:2267] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:41.168434 2024 ts_tablet_manager.cc:1905] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:41.169003 446 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:58959 (request call id 241) took 1836 ms (client timeout 1000 ms). Trace:
I20250112 02:52:41.167421 1173 ts_tablet_manager.cc:1397] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.094s user 0.005s sys 0.000s
I20250112 02:52:41.169886 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ffdc237f8640f0b378809f93fc6292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.170605 1173 raft_consensus.cc:357] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.170826 1173 raft_consensus.cc:383] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.170918 1173 raft_consensus.cc:738] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.171139 1173 consensus_queue.cc:260] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.172217 2766 tablet.cc:1721] T 856fa83dedfd4b049e3e0d91d7ac5e40 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:41.172410 1173 ts_tablet_manager.cc:1428] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 02:52:41.172672 1173 tablet_bootstrap.cc:492] T fea2ef4e9a63491db850bb5393938017 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:41.174886 1173 tablet_bootstrap.cc:654] T fea2ef4e9a63491db850bb5393938017 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.175808 2562 ts_tablet_manager.cc:1918] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:41.175951 2562 log.cc:1198] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/f924459d1150472aafb325a11825db23
I20250112 02:52:41.176337 2562 ts_tablet_manager.cc:1939] T f924459d1150472aafb325a11825db23 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:41.177150 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 238) took 1023 ms. Trace:
I20250112 02:52:41.177264 2562 rpcz_store.cc:276] 0112 02:52:40.153257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.153906 (+ 649us) service_pool.cc:225] Handling call
0112 02:52:41.017920 (+864014us) ts_tablet_manager.cc:1145] Deleting tablet f924459d1150472aafb325a11825db23
0112 02:52:41.017931 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.107993 (+ 90062us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.175725 (+ 67732us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.177133 (+ 1408us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":863863}
I20250112 02:52:41.178859 2761 tablet.cc:1721] T 856fa83dedfd4b049e3e0d91d7ac5e40 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:41.183141 2696 raft_consensus.cc:491] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.183274 2696 raft_consensus.cc:513] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.183789 2696 leader_election.cc:290] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:41.184351 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125b12a2f9164653be79d3e6d8313d36" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:41.184795 855 leader_election.cc:343] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.189399 1181 tablet_bootstrap.cc:492] T 9bbef42e11a843b18ed582ab4d04a92a P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:41.189611 1181 ts_tablet_manager.cc:1397] T 9bbef42e11a843b18ed582ab4d04a92a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.093s user 0.007s sys 0.000s
I20250112 02:52:41.190114 1177 tablet_bootstrap.cc:492] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:41.190315 2562 tablet_replica.cc:331] T 13578542e9e741e8817e0ee63b10f1a1 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:41.190543 2562 raft_consensus.cc:2238] T 13578542e9e741e8817e0ee63b10f1a1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:41.190490 1181 raft_consensus.cc:357] T 9bbef42e11a843b18ed582ab4d04a92a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.190858 1181 raft_consensus.cc:383] T 9bbef42e11a843b18ed582ab4d04a92a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.190716 2562 raft_consensus.cc:2267] T 13578542e9e741e8817e0ee63b10f1a1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:41.191449 2562 ts_tablet_manager.cc:1905] T 13578542e9e741e8817e0ee63b10f1a1 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:41.191558 1177 tablet_bootstrap.cc:654] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.190985 1181 raft_consensus.cc:738] T 9bbef42e11a843b18ed582ab4d04a92a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.191989 1181 consensus_queue.cc:260] T 9bbef42e11a843b18ed582ab4d04a92a P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.194128 1181 ts_tablet_manager.cc:1428] T 9bbef42e11a843b18ed582ab4d04a92a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:41.196308 1181 tablet_bootstrap.cc:492] T ff1b3ee229c445b19405cf4a19cf8003 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:41.176077 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a8a178bafcb457f9b0e9587df560493 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.177053 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a34d03a80ae402db6607261a5c40b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.209936 1181 tablet_bootstrap.cc:654] T ff1b3ee229c445b19405cf4a19cf8003 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.180063 2760 tablet.cc:1721] T 856fa83dedfd4b049e3e0d91d7ac5e40 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:41.214934 2766 tablet.cc:1721] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:41.218724 2762 tablet.cc:1721] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:41.219182 1173 tablet_bootstrap.cc:492] T fea2ef4e9a63491db850bb5393938017 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:41.219331 1173 ts_tablet_manager.cc:1397] T fea2ef4e9a63491db850bb5393938017 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.003s
I20250112 02:52:41.220014 1173 raft_consensus.cc:357] T fea2ef4e9a63491db850bb5393938017 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.220198 1173 raft_consensus.cc:383] T fea2ef4e9a63491db850bb5393938017 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.220276 1173 raft_consensus.cc:738] T fea2ef4e9a63491db850bb5393938017 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.220371 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125b12a2f9164653be79d3e6d8313d36" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:41.227380 2451 consensus_queue.cc:1035] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:41.220535 1173 consensus_queue.cc:260] T fea2ef4e9a63491db850bb5393938017 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.221616 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 13578542e9e741e8817e0ee63b10f1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.233222 1173 ts_tablet_manager.cc:1428] T fea2ef4e9a63491db850bb5393938017 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20250112 02:52:41.233512 1173 tablet_bootstrap.cc:492] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:41.235013 1173 tablet_bootstrap.cc:654] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.235683 2024 ts_tablet_manager.cc:1918] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:41.235793 2024 log.cc:1198] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/70715db0394d46d9b6b289160b00be7d
I20250112 02:52:41.236091 2024 ts_tablet_manager.cc:1939] T 70715db0394d46d9b6b289160b00be7d P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:41.236686 1177 tablet_bootstrap.cc:492] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:41.236814 1177 ts_tablet_manager.cc:1397] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.047s user 0.000s sys 0.003s
I20250112 02:52:41.236819 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 197) took 2331 ms. Trace:
I20250112 02:52:41.236927 2024 rpcz_store.cc:276] 0112 02:52:38.905334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.905379 (+ 45us) service_pool.cc:225] Handling call
0112 02:52:41.166956 (+2261577us) ts_tablet_manager.cc:1145] Deleting tablet 70715db0394d46d9b6b289160b00be7d
0112 02:52:41.166963 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.189098 (+ 22135us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.231658 (+ 42560us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.236799 (+ 5141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2261374}
I20250112 02:52:41.237623 1177 raft_consensus.cc:357] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.237830 1177 raft_consensus.cc:383] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.237880 1177 raft_consensus.cc:738] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.238123 1177 consensus_queue.cc:260] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.241160 1177 ts_tablet_manager.cc:1428] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20250112 02:52:41.242970 2024 tablet_replica.cc:331] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:41.243141 2024 raft_consensus.cc:2238] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:41.243410 2024 pending_rounds.cc:62] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6: Trying to abort 1 pending ops.
I20250112 02:52:41.243440 2024 pending_rounds.cc:69] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319872413638656 op_type: NO_OP noop_request { }
I20250112 02:52:41.243546 2024 raft_consensus.cc:2883] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:41.243613 2024 raft_consensus.cc:2267] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:41.244097 2024 ts_tablet_manager.cc:1905] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:41.245139 1177 tablet_bootstrap.cc:492] T e4b7c2efd7ea402e8d19038f96b9675c P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:41.246594 1177 tablet_bootstrap.cc:654] T e4b7c2efd7ea402e8d19038f96b9675c P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.251096 2024 ts_tablet_manager.cc:1918] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:41.257169 2024 log.cc:1198] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/9af55dab202444358ea1a0674d37946d
I20250112 02:52:41.247539 2451 raft_consensus.cc:491] T 3e67f0a0c065498f9a644220414415b0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.257506 2451 raft_consensus.cc:513] T 3e67f0a0c065498f9a644220414415b0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.257983 2024 ts_tablet_manager.cc:1939] T 9af55dab202444358ea1a0674d37946d P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:41.258215 2451 leader_election.cc:290] T 3e67f0a0c065498f9a644220414415b0 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:41.258607 2179 consensus_queue.cc:1035] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:41.258711 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e67f0a0c065498f9a644220414415b0" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:41.259495 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e67f0a0c065498f9a644220414415b0" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:41.259713 910 raft_consensus.cc:2388] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 289afc116aa74408aa5bff74e11c50d8 in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
I20250112 02:52:41.260454 714 leader_election.cc:304] T 3e67f0a0c065498f9a644220414415b0 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:41.260712 2179 raft_consensus.cc:3054] T 3e67f0a0c065498f9a644220414415b0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.261026 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 198) took 2342 ms. Trace:
I20250112 02:52:41.261186 2696 raft_consensus.cc:491] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.261292 2696 raft_consensus.cc:513] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.261662 2024 rpcz_store.cc:276] 0112 02:52:38.918113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.918163 (+ 50us) service_pool.cc:225] Handling call
0112 02:52:41.242926 (+2324763us) ts_tablet_manager.cc:1145] Deleting tablet 9af55dab202444358ea1a0674d37946d
0112 02:52:41.242935 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.248686 (+ 5751us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.251037 (+ 2351us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.261012 (+ 9975us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2312164}
I20250112 02:52:41.261824 2696 leader_election.cc:290] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:41.262342 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69b7a0f52304ad9951a07b3ab0372e7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:41.262754 854 leader_election.cc:343] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.263016 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69b7a0f52304ad9951a07b3ab0372e7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:41.263128 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 199) took 2342 ms. Trace:
I20250112 02:52:41.263175 2179 raft_consensus.cc:2743] T 3e67f0a0c065498f9a644220414415b0 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:41.263229 2024 rpcz_store.cc:276] 0112 02:52:38.920903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.921337 (+ 434us) service_pool.cc:225] Handling call
0112 02:52:41.263012 (+2341675us) ts_tablet_manager.cc:1145] Deleting tablet e9fc9a4f1e8944abb76dad6c2ad53d35
0112 02:52:41.263025 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.263096 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2331231}
W20250112 02:52:41.263545 855 leader_election.cc:343] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.263653 855 leader_election.cc:304] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:41.263993 2696 raft_consensus.cc:2743] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:41.264199 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 200) took 2340 ms. Trace:
I20250112 02:52:41.266860 827 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:41.267254 2696 raft_consensus.cc:491] T 9bbef42e11a843b18ed582ab4d04a92a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.267360 2696 raft_consensus.cc:513] T 9bbef42e11a843b18ed582ab4d04a92a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.267935 2696 leader_election.cc:290] T 9bbef42e11a843b18ed582ab4d04a92a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:41.268344 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbef42e11a843b18ed582ab4d04a92a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:41.268798 2024 rpcz_store.cc:276] 0112 02:52:38.923345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.923397 (+ 52us) service_pool.cc:225] Handling call
0112 02:52:41.264120 (+2340723us) ts_tablet_manager.cc:1145] Deleting tablet aba1744138fe42ae9d097db01cac083d
0112 02:52:41.264127 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.264183 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2332271}
W20250112 02:52:41.269068 854 leader_election.cc:343] T 9bbef42e11a843b18ed582ab4d04a92a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.269500 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 201) took 2342 ms. Trace:
I20250112 02:52:41.269845 2024 rpcz_store.cc:276] 0112 02:52:38.926680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.926754 (+ 74us) service_pool.cc:225] Handling call
0112 02:52:41.269416 (+2342662us) ts_tablet_manager.cc:1145] Deleting tablet 015412a5fa304453bd8d1be1c3cd7a46
0112 02:52:41.269427 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.269485 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2337552}
I20250112 02:52:41.270532 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 202) took 2341 ms. Trace:
I20250112 02:52:41.270838 2024 rpcz_store.cc:276] 0112 02:52:38.929561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.929601 (+ 40us) service_pool.cc:225] Handling call
0112 02:52:41.270443 (+2340842us) ts_tablet_manager.cc:1145] Deleting tablet d63d411f906b4d4eb68343bdcc493e05
0112 02:52:41.270451 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.270520 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2338284}
I20250112 02:52:41.271548 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 203) took 2334 ms. Trace:
I20250112 02:52:41.272485 2024 rpcz_store.cc:276] 0112 02:52:38.937133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.937193 (+ 60us) service_pool.cc:225] Handling call
0112 02:52:41.271454 (+2334261us) ts_tablet_manager.cc:1145] Deleting tablet d659c5a4f7a446c6aaf2283f65b45106
0112 02:52:41.271487 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.271534 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2334024}
I20250112 02:52:41.273051 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 204) took 2334 ms. Trace:
I20250112 02:52:41.275658 2024 rpcz_store.cc:276] 0112 02:52:38.938412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.938453 (+ 41us) service_pool.cc:225] Handling call
0112 02:52:41.272988 (+2334535us) ts_tablet_manager.cc:1145] Deleting tablet 02d0f67141dc46b6a87ca0ad080d5bfe
0112 02:52:41.272995 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.273039 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2334371}
I20250112 02:52:41.276566 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 206) took 2332 ms. Trace:
I20250112 02:52:41.276743 2024 rpcz_store.cc:276] 0112 02:52:38.944576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.944624 (+ 48us) service_pool.cc:225] Handling call
0112 02:52:41.276229 (+2331605us) ts_tablet_manager.cc:1145] Deleting tablet 65f22a7b1b804661af3b7e785a833739
0112 02:52:41.276236 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.276553 (+ 317us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2331388}
I20250112 02:52:41.277281 2024 tablet_replica.cc:331] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:41.277657 2024 raft_consensus.cc:2238] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:41.278235 2024 raft_consensus.cc:2267] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:41.279157 2024 ts_tablet_manager.cc:1905] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:41.177721 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet f924459d1150472aafb325a11825db23 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]) successfully deleted
W20250112 02:52:41.280536 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 102807us
W20250112 02:52:41.280650 854 leader_election.cc:343] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.280747 854 leader_election.cc:304] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:41.281047 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet bd5b3c712c0f448ea9b14ab93dd731ab because the tablet was not found. No further retry: Not found: Tablet not found: bd5b3c712c0f448ea9b14ab93dd731ab
I20250112 02:52:41.281141 2696 raft_consensus.cc:2743] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:41.281246 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 62f4bd2df0ce45dabc55373f1154d3ac because the tablet was not found. No further retry: Not found: Tablet not found: 62f4bd2df0ce45dabc55373f1154d3ac
W20250112 02:52:41.281407 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 7e1954e7859f422f9690fe4a9384a8bb because the tablet was not found. No further retry: Not found: Tablet not found: 7e1954e7859f422f9690fe4a9384a8bb
W20250112 02:52:41.281586 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 54215c55966a46e1ab96e352c4e3764e because the tablet was not found. No further retry: Not found: Tablet not found: 54215c55966a46e1ab96e352c4e3764e
W20250112 02:52:41.281739 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 82b351c563d749df806ab5d0331a4ef2 because the tablet was not found. No further retry: Not found: Tablet not found: 82b351c563d749df806ab5d0331a4ef2
W20250112 02:52:41.281921 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet a3ffdc237f8640f0b378809f93fc6292 because the tablet was not found. No further retry: Not found: Tablet not found: a3ffdc237f8640f0b378809f93fc6292
I20250112 02:52:41.186677 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 681a930541a84f0197ecafe670278e70
I20250112 02:52:41.282433 2562 ts_tablet_manager.cc:1918] T 13578542e9e741e8817e0ee63b10f1a1 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:41.282724 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 681a930541a84f0197ecafe670278e70 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b1d46c5395d4defb2d0e2948807eaf4) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.283162 454 catalog_manager.cc:6596] ReplaceTablet: tablet 681a930541a84f0197ecafe670278e70 deleted and replaced by tablet 6b1d46c5395d4defb2d0e2948807eaf4
I20250112 02:52:41.233603 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 8429a3d8636d4e2ca4c1191f85b9266b
I20250112 02:52:41.283915 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 940affdc2c484b7d9fd2de17f9d1cea7 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:41.284178 1177 tablet_bootstrap.cc:492] T e4b7c2efd7ea402e8d19038f96b9675c P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:41.284292 1177 ts_tablet_manager.cc:1397] T e4b7c2efd7ea402e8d19038f96b9675c P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.000s
I20250112 02:52:41.237376 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 70715db0394d46d9b6b289160b00be7d (table default.Table_bc3feb9849cf4e5db05a15330115dd52 [id=53797be8d53c45f3b805e532889be30c]) successfully deleted
W20250112 02:52:41.284945 441 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 47572.1us
I20250112 02:52:41.284878 1177 raft_consensus.cc:357] T e4b7c2efd7ea402e8d19038f96b9675c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.285048 1177 raft_consensus.cc:383] T e4b7c2efd7ea402e8d19038f96b9675c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.285104 1177 raft_consensus.cc:738] T e4b7c2efd7ea402e8d19038f96b9675c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.285295 1177 consensus_queue.cc:260] T e4b7c2efd7ea402e8d19038f96b9675c P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.285907 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 9af55dab202444358ea1a0674d37946d (table default.Table_26dfe346379948b39f90b31653c2a463 [id=6c6f30e60e79451397841d090d11d8b7]) successfully deleted
I20250112 02:52:41.285995 1177 ts_tablet_manager.cc:1428] T e4b7c2efd7ea402e8d19038f96b9675c P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250112 02:52:41.286101 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet e9fc9a4f1e8944abb76dad6c2ad53d35 because the tablet was not found. No further retry: Not found: Tablet not found: e9fc9a4f1e8944abb76dad6c2ad53d35
W20250112 02:52:41.286290 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet aba1744138fe42ae9d097db01cac083d because the tablet was not found. No further retry: Not found: Tablet not found: aba1744138fe42ae9d097db01cac083d
W20250112 02:52:41.286531 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet d63d411f906b4d4eb68343bdcc493e05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d63d411f906b4d4eb68343bdcc493e05 already in progress: creating tablet
W20250112 02:52:41.286688 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet d659c5a4f7a446c6aaf2283f65b45106 because the tablet was not found. No further retry: Not found: Tablet not found: d659c5a4f7a446c6aaf2283f65b45106
W20250112 02:52:41.286883 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 02d0f67141dc46b6a87ca0ad080d5bfe because the tablet was not found. No further retry: Not found: Tablet not found: 02d0f67141dc46b6a87ca0ad080d5bfe
W20250112 02:52:41.287066 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 65f22a7b1b804661af3b7e785a833739 because the tablet was not found. No further retry: Not found: Tablet not found: 65f22a7b1b804661af3b7e785a833739
I20250112 02:52:41.288214 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet e5c10fad790046a88fbc8ebe0303c481 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.288380 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 9163878cfeb64693a9c5493e812546af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.288502 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet ea33e5c1a1fb46599c5836a36b130051 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:25 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.288727 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ee98f055024429eafaa29bb961cab87 with delete_type TABLET_DATA_DELETED (replaced by tablet 99407ead9cf547e3bbe78c0d0b327e6a) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.289131 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet d45fce86e02f46d4a57af718fa2a62c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 5804fc4130f84c69832c1022b7bd7fa5) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.289279 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 2730634a5cfd4c2fb5c4e30100c556d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.289414 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet b9c006281dc949119951c74e23167d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.289431 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 352c87f1dc724e7b94bc2e64abfce8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.289556 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 73136be25c79491784d3ea71aa9b27f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.289582 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 8429a3d8636d4e2ca4c1191f85b9266b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.289717 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c639a2bcd4740e4be7b2b162004669a) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.289831 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 681a930541a84f0197ecafe670278e70 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b1d46c5395d4defb2d0e2948807eaf4) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.289880 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b599e983ec04f53a440d1ab828d12ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:26 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.289950 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet b61f16f3d601436bb4b32344de28c1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.289980 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet d4abf59d96414e5faa7cf75ba133832c with delete_type TABLET_DATA_DELETED (replaced by tablet 5a3af0ea4e954053ae89814f3e74e6b7) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.283277 2696 raft_consensus.cc:491] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.290055 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet cbe30f9860e2492b90e8cedf8ff9daba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.290097 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a12b225131b464494a9f02133d37aea with delete_type TABLET_DATA_DELETED (replaced by tablet d45fce86e02f46d4a57af718fa2a62c8) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.290225 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet f05c8426ea3d4fe89fbe6c88c688019f with delete_type TABLET_DATA_DELETED (replaced by tablet c5079f87652a4ea58b366d1f9448fd4e) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.290140 2696 raft_consensus.cc:513] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.290235 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet bb217afc08304c0ea35dee3120347ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.290360 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet daef3a58605e40978f7389698b22265d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.290496 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet a12c5d58e6c246c983e8a533906ce77d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.290612 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 003a3b4705e943c6a900762ec11ddd47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.290690 2696 leader_election.cc:290] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:41.291239 2562 log.cc:1198] T 13578542e9e741e8817e0ee63b10f1a1 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/13578542e9e741e8817e0ee63b10f1a1
I20250112 02:52:41.291688 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 8429a3d8636d4e2ca4c1191f85b9266b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.292407 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c04f1a2cb04950a9f6170e252c19b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.293175 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:41.293373 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 681a930541a84f0197ecafe670278e70 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b1d46c5395d4defb2d0e2948807eaf4) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:41.293370 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 8429a3d8636d4e2ca4c1191f85b9266b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:41.297524 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_4770176caf32467daca85ee46a034acd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:41.298157 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4770176caf32467daca85ee46a034acd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:41.298745 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c29a9ca43e034b43a00207e5847bbee3" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:41.299243 854 leader_election.cc:343] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.299350 2562 ts_tablet_manager.cc:1939] T 13578542e9e741e8817e0ee63b10f1a1 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:41.301308 2745 raft_consensus.cc:491] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.301411 2745 raft_consensus.cc:513] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.301955 2745 leader_election.cc:290] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:41.302570 2745 raft_consensus.cc:491] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.302570 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e477fc157aed4563923d85d66014293a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:41.302739 910 raft_consensus.cc:2463] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:41.302685 2745 raft_consensus.cc:513] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.303092 989 leader_election.cc:304] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:41.303202 2745 leader_election.cc:290] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:41.303319 2228 raft_consensus.cc:2798] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:41.303387 2228 raft_consensus.cc:491] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:41.302966 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 246) took 1078 ms. Trace:
I20250112 02:52:41.303452 2228 raft_consensus.cc:3054] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.304286 2024 ts_tablet_manager.cc:1918] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:41.304376 2024 log.cc:1198] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/416fb6820cd54e118950c68d381f51b2
I20250112 02:52:41.303455 2562 rpcz_store.cc:276] 0112 02:52:40.224949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.224989 (+ 40us) service_pool.cc:225] Handling call
0112 02:52:41.190275 (+965286us) ts_tablet_manager.cc:1145] Deleting tablet 13578542e9e741e8817e0ee63b10f1a1
0112 02:52:41.190283 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.215025 (+ 24742us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.282336 (+ 67311us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.302948 (+ 20612us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":965138}
I20250112 02:52:41.305094 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 13578542e9e741e8817e0ee63b10f1a1 (table default.Table_63abe3ac694a4ed6bf4aef720b542e73 [id=a7ce19c248fe4ac6a6b436773d6c1b4c]) successfully deleted
I20250112 02:52:41.305737 2228 raft_consensus.cc:513] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.306234 2228 leader_election.cc:290] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:41.306744 2024 ts_tablet_manager.cc:1939] T 416fb6820cd54e118950c68d381f51b2 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:41.307255 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777272e7159b438ebff4db93698411d0" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:41.307513 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e477fc157aed4563923d85d66014293a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:41.308300 910 raft_consensus.cc:3054] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.308427 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 207) took 2361 ms. Trace:
I20250112 02:52:41.308496 2024 rpcz_store.cc:276] 0112 02:52:38.946579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.946630 (+ 51us) service_pool.cc:225] Handling call
0112 02:52:41.277238 (+2330608us) ts_tablet_manager.cc:1145] Deleting tablet 416fb6820cd54e118950c68d381f51b2
0112 02:52:41.277247 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.294249 (+ 17002us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.300395 (+ 6146us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.308412 (+ 8017us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2330377}
I20250112 02:52:41.308779 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 247) took 1079 ms. Trace:
I20250112 02:52:41.308881 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 416fb6820cd54e118950c68d381f51b2 (table default.Table_f4d612cd6a524bc493147493c5a4a284 [id=4ac16bc62bbe4ed7a977d22c92ad580d]) successfully deleted
I20250112 02:52:41.308907 2562 rpcz_store.cc:276] 0112 02:52:40.229671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.229712 (+ 41us) service_pool.cc:225] Handling call
0112 02:52:41.305241 (+1075529us) ts_tablet_manager.cc:1145] Deleting tablet d63d411f906b4d4eb68343bdcc493e05
0112 02:52:41.308644 (+ 3403us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.308738 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1075235}
I20250112 02:52:41.309525 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 248) took 1079 ms. Trace:
I20250112 02:52:41.310140 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 208) took 2359 ms. Trace:
I20250112 02:52:41.310206 2024 rpcz_store.cc:276] 0112 02:52:38.950445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.950487 (+ 42us) service_pool.cc:225] Handling call
0112 02:52:41.310037 (+2359550us) ts_tablet_manager.cc:1145] Deleting tablet 7e1954e7859f422f9690fe4a9384a8bb
0112 02:52:41.310047 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.310126 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2359329}
W20250112 02:52:41.310545 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 7e1954e7859f422f9690fe4a9384a8bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e1954e7859f422f9690fe4a9384a8bb already in progress: creating tablet
I20250112 02:52:41.309751 2562 rpcz_store.cc:276] 0112 02:52:40.230412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.230452 (+ 40us) service_pool.cc:225] Handling call
0112 02:52:41.309387 (+1078935us) ts_tablet_manager.cc:1145] Deleting tablet cbe30f9860e2492b90e8cedf8ff9daba
0112 02:52:41.309395 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.309512 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1078806}
I20250112 02:52:41.311489 910 raft_consensus.cc:2463] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
W20250112 02:52:41.311805 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet d63d411f906b4d4eb68343bdcc493e05 because the tablet was not found. No further retry: Not found: Tablet not found: d63d411f906b4d4eb68343bdcc493e05
I20250112 02:52:41.311800 989 leader_election.cc:304] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
W20250112 02:52:41.312016 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet cbe30f9860e2492b90e8cedf8ff9daba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbe30f9860e2492b90e8cedf8ff9daba already in progress: creating tablet
I20250112 02:52:41.312013 2228 raft_consensus.cc:2798] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:41.312153 2228 raft_consensus.cc:695] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:41.312386 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 209) took 2359 ms. Trace:
I20250112 02:52:41.312433 917 raft_consensus.cc:2463] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:41.307835 1181 tablet_bootstrap.cc:492] T ff1b3ee229c445b19405cf4a19cf8003 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:41.312784 989 leader_election.cc:304] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:41.312824 2024 rpcz_store.cc:276] 0112 02:52:38.952434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.952472 (+ 38us) service_pool.cc:225] Handling call
0112 02:52:41.312310 (+2359838us) ts_tablet_manager.cc:1145] Deleting tablet 161fe26b9c0e4543b7aeb0021506d72a
0112 02:52:41.312319 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.312372 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2359668}
I20250112 02:52:41.312942 1181 ts_tablet_manager.cc:1397] T ff1b3ee229c445b19405cf4a19cf8003 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.118s user 0.007s sys 0.001s
I20250112 02:52:41.313050 2745 raft_consensus.cc:2798] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:41.313210 2745 raft_consensus.cc:491] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:41.313293 2745 raft_consensus.cc:3054] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.313395 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 249) took 1077 ms. Trace:
I20250112 02:52:41.314114 1181 raft_consensus.cc:357] T ff1b3ee229c445b19405cf4a19cf8003 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.314420 1181 raft_consensus.cc:383] T ff1b3ee229c445b19405cf4a19cf8003 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.314462 1181 raft_consensus.cc:738] T ff1b3ee229c445b19405cf4a19cf8003 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.315232 1181 consensus_queue.cc:260] T ff1b3ee229c445b19405cf4a19cf8003 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.312364 2228 consensus_queue.cc:237] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.315711 2326 consensus_queue.cc:1035] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:41.315984 1181 ts_tablet_manager.cc:1428] T ff1b3ee229c445b19405cf4a19cf8003 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250112 02:52:41.315318 2745 raft_consensus.cc:513] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.316367 1181 tablet_bootstrap.cc:492] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:41.316490 2745 leader_election.cc:290] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:41.317627 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777272e7159b438ebff4db93698411d0" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:41.317799 910 raft_consensus.cc:3054] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.313755 2562 rpcz_store.cc:276] 0112 02:52:40.235857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.235907 (+ 50us) service_pool.cc:225] Handling call
0112 02:52:41.312113 (+1076206us) ts_tablet_manager.cc:1145] Deleting tablet 83c04f1a2cb04950a9f6170e252c19b4
0112 02:52:41.312130 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.312961 (+ 831us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1075744}
I20250112 02:52:41.320117 910 raft_consensus.cc:2463] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
W20250112 02:52:41.320333 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 83c04f1a2cb04950a9f6170e252c19b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83c04f1a2cb04950a9f6170e252c19b4 already in progress: creating tablet
I20250112 02:52:41.320835 989 leader_election.cc:304] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:41.321106 2745 raft_consensus.cc:2798] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:41.321364 2745 raft_consensus.cc:695] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:41.321645 2745 consensus_queue.cc:237] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.325500 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 250) took 1088 ms. Trace:
I20250112 02:52:41.325600 2562 rpcz_store.cc:276] 0112 02:52:40.236601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.236641 (+ 40us) service_pool.cc:225] Handling call
0112 02:52:41.325345 (+1088704us) ts_tablet_manager.cc:1145] Deleting tablet b6d84027ad6d40a5be5afcdd854431a5
0112 02:52:41.325362 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.325483 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1082580}
W20250112 02:52:41.326023 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet b6d84027ad6d40a5be5afcdd854431a5 because the tablet was not found. No further retry: Not found: Tablet not found: b6d84027ad6d40a5be5afcdd854431a5
I20250112 02:52:41.326079 2562 tablet_replica.cc:331] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:41.326292 2562 raft_consensus.cc:2238] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:41.326514 2562 raft_consensus.cc:2267] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:41.327382 2562 ts_tablet_manager.cc:1905] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:41.331677 2381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=522 overwritten=0 applied=513 ignored=0} inserts{seen=473 ignored=0} mutations{seen=583 ignored=0} orphaned_commits=0. Pending: 9 replicates
I20250112 02:52:41.332543 2381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Bootstrap complete.
I20250112 02:52:41.334646 1181 tablet_bootstrap.cc:654] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.338361 897 tablet_service.cc:1514] Processing DeleteTablet for tablet f924459d1150472aafb325a11825db23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:39 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:41.338766 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c04f1a2cb04950a9f6170e252c19b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
W20250112 02:52:41.304899 446 rpcz_store.cc:269] 0112 02:52:39.332584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.345271 (+ 12687us) service_pool.cc:225] Handling call
0112 02:52:39.862353 (+517082us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.862359 (+ 6us) write_op.cc:454] Released schema lock
0112 02:52:39.926938 (+ 64579us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.926943 (+ 5us) write_op.cc:454] Released schema lock
0112 02:52:41.168935 (+1241992us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:39.855578 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:39.855666 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:39.855671 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:39.855674 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:39.855857 (+ 183us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:39.855868 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:39.855872 (+ 4us) tablet.cc:682] Acquiring locks for 7 operations
0112 02:52:39.855992 (+ 120us) tablet.cc:698] Row locks acquired
0112 02:52:39.855994 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:39.856024 (+ 30us) write_op.cc:270] Start()
0112 02:52:39.856052 (+ 28us) write_op.cc:276] Timestamp: P: 1736650359855967 usec, L: 0
0112 02:52:39.856055 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:39.856233 (+ 178us) log.cc:844] Serialized 2772 byte log entry
0112 02:52:39.859576 (+ 3343us) op_driver.cc:464] REPLICATION: finished
0112 02:52:39.859682 (+ 106us) write_op.cc:301] APPLY: starting
0112 02:52:39.859696 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:39.859800 (+ 104us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:39.859826 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:39.859942 (+ 116us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:39.859983 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0112 02:52:39.859990 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:39.860062 (+ 72us) log.cc:844] Serialized 59 byte log entry
0112 02:52:39.860117 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.860136 (+ 19us) write_op.cc:454] Released schema lock
0112 02:52:39.860143 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 02:52:39.921691 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:39.921806 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:39.921812 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:39.921815 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:39.921913 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:39.921925 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:39.921930 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:39.921981 (+ 51us) tablet.cc:698] Row locks acquired
0112 02:52:39.921984 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:39.922014 (+ 30us) write_op.cc:270] Start()
0112 02:52:39.922041 (+ 27us) write_op.cc:276] Timestamp: P: 1736650359921943 usec, L: 0
0112 02:52:39.922045 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:39.922207 (+ 162us) log.cc:844] Serialized 356 byte log entry
0112 02:52:39.925376 (+ 3169us) op_driver.cc:464] REPLICATION: finished
0112 02:52:39.925507 (+ 131us) write_op.cc:301] APPLY: starting
0112 02:52:39.925519 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:39.925555 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:39.925578 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:39.925617 (+ 39us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:39.925686 (+ 69us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:39.925693 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:39.925766 (+ 73us) log.cc:844] Serialized 23 byte log entry
0112 02:52:39.925812 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:39.925828 (+ 16us) write_op.cc:454] Released schema lock
0112 02:52:39.925833 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 02:52:40.184190 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.184298 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.184305 (+ 7us) write_op.cc:435] Acquired schema lock
0112 02:52:40.184309 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:40.184395 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.184404 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.184408 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:40.184458 (+ 50us) tablet.cc:698] Row locks acquired
0112 02:52:40.184460 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:40.184489 (+ 29us) write_op.cc:270] Start()
0112 02:52:40.184519 (+ 30us) write_op.cc:276] Timestamp: P: 1736650360184468 usec, L: 0
0112 02:52:40.184523 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.184654 (+ 131us) log.cc:844] Serialized 356 byte log entry
0112 02:52:40.199536 (+ 14882us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.199938 (+ 402us) write_op.cc:301] APPLY: starting
0112 02:52:40.199970 (+ 32us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.200001 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.200021 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.200060 (+ 39us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.200084 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:40.200090 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:40.200165 (+ 75us) log.cc:844] Serialized 23 byte log entry
0112 02:52:40.200217 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.200230 (+ 13us) write_op.cc:454] Released schema lock
0112 02:52:40.200237 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:40.200298 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.200302 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2921,"reported_tablets":172,"child_traces":[["op",{"apply.queue_time_us":62,"apply.run_cpu_time_us":511,"apply.run_wall_time_us":511,"num_ops":7,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":746,"prepare.run_wall_time_us":746,"raft.queue_time_us":54,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":226,"replication_time_us":3495}],["op",{"apply.queue_time_us":90,"apply.run_cpu_time_us":369,"apply.run_wall_time_us":369,"num_ops":1,"prepare.queue_time_us":239,"prepare.run_cpu_time_us":614,"prepare.run_wall_time_us":614,"raft.queue_time_us":80,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":204,"replication_time_us":3307}],["op",{"apply.queue_time_us":355,"apply.run_cpu_time_us":354,"apply.run_wall_time_us":354,"num_ops":1,"prepare.queue_time_us":881,"prepare.run_cpu_time_us":561,"prepare.run_wall_time_us":5252,"replication_time_us":14991}]]}
I20250112 02:52:41.347311 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_9fdbd960fe3a41c59ab9262ef27fcf95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:41.348215 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 161fe26b9c0e4543b7aeb0021506d72a because the tablet was not found. No further retry: Not found: Tablet not found: 161fe26b9c0e4543b7aeb0021506d72a
W20250112 02:52:41.348464 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9fdbd960fe3a41c59ab9262ef27fcf95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:41.348822 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 210) took 2395 ms. Trace:
I20250112 02:52:41.348959 2024 rpcz_store.cc:276] 0112 02:52:38.953449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.953485 (+ 36us) service_pool.cc:225] Handling call
0112 02:52:41.348677 (+2395192us) ts_tablet_manager.cc:1145] Deleting tablet afa2f8aabe5e42ccbc15b193fdc03563
0112 02:52:41.348688 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.348803 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2394964}
I20250112 02:52:41.349854 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e477fc157aed4563923d85d66014293a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:41.350358 1177 tablet_bootstrap.cc:492] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:41.350507 991 leader_election.cc:343] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:41.351078 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet afa2f8aabe5e42ccbc15b193fdc03563 because the tablet was not found. No further retry: Not found: Tablet not found: afa2f8aabe5e42ccbc15b193fdc03563
I20250112 02:52:41.351841 1177 tablet_bootstrap.cc:654] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.352644 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777272e7159b438ebff4db93698411d0" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:41.352857 778 raft_consensus.cc:2463] T 777272e7159b438ebff4db93698411d0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:41.353152 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbef42e11a843b18ed582ab4d04a92a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:41.353583 855 leader_election.cc:343] T 9bbef42e11a843b18ed582ab4d04a92a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.353680 855 leader_election.cc:304] T 9bbef42e11a843b18ed582ab4d04a92a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:41.354094 2696 raft_consensus.cc:2743] T 9bbef42e11a843b18ed582ab4d04a92a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:41.350369 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c29a9ca43e034b43a00207e5847bbee3" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:41.354719 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e477fc157aed4563923d85d66014293a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:41.355134 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777272e7159b438ebff4db93698411d0" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:41.355265 774 raft_consensus.cc:3054] T 777272e7159b438ebff4db93698411d0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.357990 2326 consensus_queue.cc:1035] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:41.358382 774 raft_consensus.cc:2463] T 777272e7159b438ebff4db93698411d0 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:41.358762 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 211) took 2403 ms. Trace:
I20250112 02:52:41.358951 2024 rpcz_store.cc:276] 0112 02:52:38.955684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.955746 (+ 62us) service_pool.cc:225] Handling call
0112 02:52:41.358607 (+2402861us) ts_tablet_manager.cc:1145] Deleting tablet b9199b0d2b924a79938a979df4196bb4
0112 02:52:41.358641 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.358723 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2402662}
W20250112 02:52:41.359308 991 leader_election.cc:343] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:41.359627 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b9199b0d2b924a79938a979df4196bb4 because the tablet was not found. No further retry: Not found: Tablet not found: b9199b0d2b924a79938a979df4196bb4
I20250112 02:52:41.360142 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 213) took 2394 ms. Trace:
I20250112 02:52:41.360337 2024 rpcz_store.cc:276] 0112 02:52:38.965981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.966018 (+ 37us) service_pool.cc:225] Handling call
0112 02:52:41.359994 (+2393976us) ts_tablet_manager.cc:1145] Deleting tablet cb724feccc7f4fbc846fb6fa61be3385
0112 02:52:41.360007 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.360124 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2393753}
W20250112 02:52:41.360930 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet cb724feccc7f4fbc846fb6fa61be3385 because the tablet was not found. No further retry: Not found: Tablet not found: cb724feccc7f4fbc846fb6fa61be3385
W20250112 02:52:41.361214 855 leader_election.cc:343] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.361377 855 leader_election.cc:304] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:41.361424 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 217) took 2390 ms. Trace:
W20250112 02:52:41.361555 855 raft_consensus.cc:2681] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 02:52:41.361547 2024 rpcz_store.cc:276] 0112 02:52:38.970884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.970947 (+ 63us) service_pool.cc:225] Handling call
0112 02:52:41.361231 (+2390284us) ts_tablet_manager.cc:1145] Deleting tablet c7c530aa0aa64e47ae27d61f341d2bd9
0112 02:52:41.361285 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.361386 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2390103}
W20250112 02:52:41.362126 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet c7c530aa0aa64e47ae27d61f341d2bd9 because the tablet was not found. No further retry: Not found: Tablet not found: c7c530aa0aa64e47ae27d61f341d2bd9
W20250112 02:52:41.362635 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 681a930541a84f0197ecafe670278e70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 681a930541a84f0197ecafe670278e70 already in progress: creating tablet
W20250112 02:52:41.362952 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 8429a3d8636d4e2ca4c1191f85b9266b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8429a3d8636d4e2ca4c1191f85b9266b already in progress: creating tablet
I20250112 02:52:41.362589 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 216) took 2391 ms. Trace:
I20250112 02:52:41.368064 1181 tablet_bootstrap.cc:492] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:41.368248 1181 ts_tablet_manager.cc:1397] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
I20250112 02:52:41.369127 1181 raft_consensus.cc:357] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.369411 1181 raft_consensus.cc:383] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.369518 1181 raft_consensus.cc:738] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.369794 1181 consensus_queue.cc:260] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.370915 1181 ts_tablet_manager.cc:1428] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:41.371495 1181 tablet_bootstrap.cc:492] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:41.381574 1173 tablet_bootstrap.cc:492] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:41.366945 2024 rpcz_store.cc:276] 0112 02:52:38.970749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.971121 (+ 372us) service_pool.cc:225] Handling call
0112 02:52:41.362446 (+2391325us) ts_tablet_manager.cc:1145] Deleting tablet 1a1e3af2883341e9b957f6ed748e604d
0112 02:52:41.362460 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.362547 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2391127}
I20250112 02:52:41.387614 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 218) took 2416 ms. Trace:
W20250112 02:52:41.387640 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 1a1e3af2883341e9b957f6ed748e604d because the tablet was not found. No further retry: Not found: Tablet not found: 1a1e3af2883341e9b957f6ed748e604d
I20250112 02:52:41.387724 2024 rpcz_store.cc:276] 0112 02:52:38.970983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.971149 (+ 166us) service_pool.cc:225] Handling call
0112 02:52:41.387514 (+2416365us) ts_tablet_manager.cc:1145] Deleting tablet 1726d61cc36a4deb83c5d604e8daf4f0
0112 02:52:41.387529 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.387597 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26,"tablet-delete.queue_time_us":2416187}
I20250112 02:52:41.388204 1177 tablet_bootstrap.cc:492] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:41.388257 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 219) took 2417 ms. Trace:
I20250112 02:52:41.388337 2024 rpcz_store.cc:276] 0112 02:52:38.971076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.971330 (+ 254us) service_pool.cc:225] Handling call
0112 02:52:41.388183 (+2416853us) ts_tablet_manager.cc:1145] Deleting tablet f7cd0927c6a4401cbe69f55d4db9963a
0112 02:52:41.388191 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.388243 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2416709}
I20250112 02:52:41.388700 1177 ts_tablet_manager.cc:1397] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.009s
W20250112 02:52:41.388717 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f7cd0927c6a4401cbe69f55d4db9963a because the tablet was not found. No further retry: Not found: Tablet not found: f7cd0927c6a4401cbe69f55d4db9963a
I20250112 02:52:41.388772 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 215) took 2418 ms. Trace:
I20250112 02:52:41.388870 2024 rpcz_store.cc:276] 0112 02:52:38.970659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.972475 (+ 1816us) service_pool.cc:225] Handling call
0112 02:52:41.388699 (+2416224us) ts_tablet_manager.cc:1145] Deleting tablet b7bd96fb381f4e73a0081be3fc799c36
0112 02:52:41.388707 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.388758 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2416075}
W20250112 02:52:41.389209 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b7bd96fb381f4e73a0081be3fc799c36 because the tablet was not found. No further retry: Not found: Tablet not found: b7bd96fb381f4e73a0081be3fc799c36
I20250112 02:52:41.389278 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 225) took 2404 ms. Trace:
I20250112 02:52:41.389343 2024 rpcz_store.cc:276] 0112 02:52:38.984740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.984783 (+ 43us) service_pool.cc:225] Handling call
0112 02:52:41.389186 (+2404403us) ts_tablet_manager.cc:1145] Deleting tablet 54215c55966a46e1ab96e352c4e3764e
0112 02:52:41.389194 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.389265 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2404207}
I20250112 02:52:41.389410 965 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
W20250112 02:52:41.389668 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 54215c55966a46e1ab96e352c4e3764e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54215c55966a46e1ab96e352c4e3764e already in progress: creating tablet
I20250112 02:52:41.389714 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 227) took 2397 ms. Trace:
I20250112 02:52:41.389631 1177 raft_consensus.cc:357] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.389823 1177 raft_consensus.cc:383] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.389776 2024 rpcz_store.cc:276] 0112 02:52:38.992073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.992117 (+ 44us) service_pool.cc:225] Handling call
0112 02:52:41.389641 (+2397524us) ts_tablet_manager.cc:1145] Deleting tablet aadb2ab1d2024f889841ac91e065e064
0112 02:52:41.389651 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.389702 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2397357}
W20250112 02:52:41.390153 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet aadb2ab1d2024f889841ac91e065e064 because the tablet was not found. No further retry: Not found: Tablet not found: aadb2ab1d2024f889841ac91e065e064
I20250112 02:52:41.390172 2696 raft_consensus.cc:491] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.391908 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 228) took 2399 ms. Trace:
I20250112 02:52:41.392020 2024 rpcz_store.cc:276] 0112 02:52:38.992408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.992450 (+ 42us) service_pool.cc:225] Handling call
0112 02:52:41.391821 (+2399371us) ts_tablet_manager.cc:1145] Deleting tablet a1a0a13696fc41ccb28218f1275a4b5d
0112 02:52:41.391828 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.391894 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2399199}
W20250112 02:52:41.392500 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet a1a0a13696fc41ccb28218f1275a4b5d because the tablet was not found. No further retry: Not found: Tablet not found: a1a0a13696fc41ccb28218f1275a4b5d
I20250112 02:52:41.393254 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 229) took 2400 ms. Trace:
I20250112 02:52:41.393359 2024 rpcz_store.cc:276] 0112 02:52:38.992863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.992899 (+ 36us) service_pool.cc:225] Handling call
0112 02:52:41.393171 (+2400272us) ts_tablet_manager.cc:1145] Deleting tablet 41e166b3f8414ababbd4a3f07fb90c4c
0112 02:52:41.393179 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.393240 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2400109}
W20250112 02:52:41.394316 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 41e166b3f8414ababbd4a3f07fb90c4c because the tablet was not found. No further retry: Not found: Tablet not found: 41e166b3f8414ababbd4a3f07fb90c4c
I20250112 02:52:41.394598 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 230) took 2401 ms. Trace:
I20250112 02:52:41.394695 2024 rpcz_store.cc:276] 0112 02:52:38.993166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.993277 (+ 111us) service_pool.cc:225] Handling call
0112 02:52:41.394532 (+2401255us) ts_tablet_manager.cc:1145] Deleting tablet a47cdc208fbc431b8326364e84432cf5
0112 02:52:41.394539 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.394585 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2401132}
W20250112 02:52:41.394857 450 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:58959 (request call id 242) took 1008 ms (client timeout 1000 ms). Trace:
I20250112 02:52:41.395380 2024 tablet_replica.cc:331] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:41.395761 2024 raft_consensus.cc:2238] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:41.395967 2696 raft_consensus.cc:513] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.396522 2024 raft_consensus.cc:2267] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:41.396526 895 tablet_service.cc:1514] Processing DeleteTablet for tablet cbe30f9860e2492b90e8cedf8ff9daba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:41.396977 2745 consensus_queue.cc:1035] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:41.397085 2024 ts_tablet_manager.cc:1905] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:41.391316 2381 raft_consensus.cc:357] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 3 FOLLOWER]: Replica starting. Triggering 9 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:41.407547 1173 ts_tablet_manager.cc:1397] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.174s user 0.011s sys 0.027s
I20250112 02:52:41.408385 1177 raft_consensus.cc:738] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.408653 1177 consensus_queue.cc:260] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.409286 1177 ts_tablet_manager.cc:1428] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.020s user 0.000s sys 0.001s
I20250112 02:52:41.409565 1177 tablet_bootstrap.cc:492] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:41.414857 1173 raft_consensus.cc:357] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.416118 1173 raft_consensus.cc:383] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.194931 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 76009bac22ce4136b148b493a004a0f7
I20250112 02:52:41.417171 447 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40610 (request call id 171) took 1054 ms. Trace:
W20250112 02:52:41.395174 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet a47cdc208fbc431b8326364e84432cf5 because the tablet was not found. No further retry: Not found: Tablet not found: a47cdc208fbc431b8326364e84432cf5
I20250112 02:52:41.418114 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 76009bac22ce4136b148b493a004a0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.418915 1177 tablet_bootstrap.cc:654] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.402096 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 940affdc2c484b7d9fd2de17f9d1cea7
W20250112 02:52:41.280267 451 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:38361 (request call id 213) took 2825 ms (client timeout 1000 ms). Trace:
I20250112 02:52:41.419375 1173 raft_consensus.cc:738] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.420367 826 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:41.420823 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 76009bac22ce4136b148b493a004a0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:41.421577 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 70715db0394d46d9b6b289160b00be7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.421780 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 76009bac22ce4136b148b493a004a0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.422744 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 54215c55966a46e1ab96e352c4e3764e with delete_type TABLET_DATA_DELETED (replaced by tablet a470f84d50df4cf0b4a1e5d248b4c32a) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.422940 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d411f906b4d4eb68343bdcc493e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.423115 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1954e7859f422f9690fe4a9384a8bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.423296 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 416fb6820cd54e118950c68d381f51b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.423506 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af55dab202444358ea1a0674d37946d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.428269 2673 raft_consensus.cc:491] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.428406 2673 raft_consensus.cc:513] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.429045 2673 leader_election.cc:290] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:41.429569 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea2ef4e9a63491db850bb5393938017" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:41.430153 855 leader_election.cc:343] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.430611 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea2ef4e9a63491db850bb5393938017" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:41.430778 1053 raft_consensus.cc:2463] T fea2ef4e9a63491db850bb5393938017 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:41.431263 854 leader_election.cc:304] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:41.431659 2673 raft_consensus.cc:2798] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:41.431725 2673 raft_consensus.cc:491] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:41.431779 2673 raft_consensus.cc:3054] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:41.434449 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 76009bac22ce4136b148b493a004a0f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76009bac22ce4136b148b493a004a0f7 already in progress: creating tablet
I20250112 02:52:41.434417 2673 raft_consensus.cc:513] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.435111 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 940affdc2c484b7d9fd2de17f9d1cea7 with delete_type TABLET_DATA_DELETED (replaced by tablet ab183c43f6a64270ad744e61364f0a99) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.436091 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea2ef4e9a63491db850bb5393938017" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
W20250112 02:52:41.436582 855 leader_election.cc:343] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.437587 2673 leader_election.cc:290] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:41.438287 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea2ef4e9a63491db850bb5393938017" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:41.438444 1053 raft_consensus.cc:3054] T fea2ef4e9a63491db850bb5393938017 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.435806 447 rpcz_store.cc:276] 0112 02:52:40.362572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.082643 (+720071us) service_pool.cc:225] Handling call
0112 02:52:41.083205 (+ 562us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:41.083350 (+ 145us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:52:41.083414 (+ 64us) catalog_manager.cc:2762] Locking tablets
0112 02:52:41.191007 (+107593us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:52:41.194721 (+ 3714us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:52:41.194822 (+ 101us) catalog_manager.cc:2810] Committing in-memory state
0112 02:52:41.194872 (+ 50us) catalog_manager.cc:2814] Aborting table tasks
0112 02:52:41.417120 (+222248us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:41.191552 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.191630 (+ 78us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.191634 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:41.191636 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:41.191719 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.191729 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.191733 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:41.191835 (+ 102us) tablet.cc:698] Row locks acquired
0112 02:52:41.191837 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:41.191863 (+ 26us) write_op.cc:270] Start()
0112 02:52:41.191893 (+ 30us) write_op.cc:276] Timestamp: P: 1736650361191845 usec, L: 0
0112 02:52:41.191896 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:41.192037 (+ 141us) log.cc:844] Serialized 725 byte log entry
0112 02:52:41.194182 (+ 2145us) op_driver.cc:464] REPLICATION: finished
0112 02:52:41.194281 (+ 99us) write_op.cc:301] APPLY: starting
0112 02:52:41.194292 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:41.194325 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:41.194344 (+ 19us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:41.194397 (+ 53us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:41.194425 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:41.194434 (+ 9us) write_op.cc:312] APPLY: finished
0112 02:52:41.194489 (+ 55us) log.cc:844] Serialized 29 byte log entry
0112 02:52:41.194540 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.194555 (+ 15us) write_op.cc:454] Released schema lock
0112 02:52:41.194561 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:41.194620 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.194624 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":42,"apply.run_cpu_time_us":332,"apply.run_wall_time_us":332,"num_ops":2,"prepare.queue_time_us":38,"prepare.run_cpu_time_us":568,"prepare.run_wall_time_us":568,"raft.queue_time_us":82,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":216,"replication_time_us":2261}]]}
I20250112 02:52:41.440958 1053 raft_consensus.cc:2463] T fea2ef4e9a63491db850bb5393938017 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:41.442070 854 leader_election.cc:304] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:41.443492 2673 raft_consensus.cc:2798] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:41.446892 2673 raft_consensus.cc:695] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:41.447157 2673 consensus_queue.cc:237] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.450062 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 940affdc2c484b7d9fd2de17f9d1cea7 with delete_type TABLET_DATA_DELETED (replaced by tablet ab183c43f6a64270ad744e61364f0a99) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.451040 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 940affdc2c484b7d9fd2de17f9d1cea7 with delete_type TABLET_DATA_DELETED (replaced by tablet ab183c43f6a64270ad744e61364f0a99) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:41.451596 455 catalog_manager.cc:6596] ReplaceTablet: tablet 940affdc2c484b7d9fd2de17f9d1cea7 deleted and replaced by tablet ab183c43f6a64270ad744e61364f0a99
I20250112 02:52:41.460191 2381 raft_consensus.cc:738] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4a49a5756b143fcb767c3f41463f8d9, State: Initialized, Role: FOLLOWER
W20250112 02:52:41.462025 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet d95f98ae7bdf42518bbff55e257fea88 (table default.Table_2fcdbb05970f4a098f7ba3e287fa0348 [id=70c61753e9944e1fbd65e9ec41a48ee2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d95f98ae7bdf42518bbff55e257fea88 consensus state has no leader
I20250112 02:52:41.463235 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 13578542e9e741e8817e0ee63b10f1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:41.463557 2696 leader_election.cc:290] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:41.464344 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3d291ca3494f05ac23c7fe293e0bd9" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:41.464814 854 leader_election.cc:343] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.465520 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3d291ca3494f05ac23c7fe293e0bd9" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:41.466053 855 leader_election.cc:343] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.466799 2381 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 513, Last appended: 3.522, Last appended by leader: 522, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:41.471565 2696 raft_consensus.cc:491] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 02:52:41.472020 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 940affdc2c484b7d9fd2de17f9d1cea7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 940affdc2c484b7d9fd2de17f9d1cea7 already in progress: creating tablet
I20250112 02:52:41.466158 855 leader_election.cc:304] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:41.471668 2696 raft_consensus.cc:513] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.474900 2562 ts_tablet_manager.cc:1918] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:41.474995 2562 log.cc:1198] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/c29a9ca43e034b43a00207e5847bbee3
I20250112 02:52:41.475270 2562 ts_tablet_manager.cc:1939] T c29a9ca43e034b43a00207e5847bbee3 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:41.477775 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 251) took 1240 ms. Trace:
I20250112 02:52:41.477874 2562 rpcz_store.cc:276] 0112 02:52:40.237005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.237053 (+ 48us) service_pool.cc:225] Handling call
0112 02:52:41.326039 (+1088986us) ts_tablet_manager.cc:1145] Deleting tablet c29a9ca43e034b43a00207e5847bbee3
0112 02:52:41.326049 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.380027 (+ 53978us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.474806 (+ 94779us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.477762 (+ 2956us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1088838}
W20250112 02:52:41.481531 451 rpcz_store.cc:269] 0112 02:52:38.454746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.079627 (+624881us) service_pool.cc:225] Handling call
0112 02:52:41.280215 (+2200588us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:40.015381 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.015495 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.015503 (+ 8us) write_op.cc:435] Acquired schema lock
0112 02:52:40.015507 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:40.015627 (+ 120us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.015643 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.015650 (+ 7us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:40.015718 (+ 68us) tablet.cc:698] Row locks acquired
0112 02:52:40.015723 (+ 5us) write_op.cc:260] PREPARE: finished
0112 02:52:40.015763 (+ 40us) write_op.cc:270] Start()
0112 02:52:40.015800 (+ 37us) write_op.cc:276] Timestamp: P: 1736650360015669 usec, L: 0
0112 02:52:40.015805 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.016012 (+ 207us) log.cc:844] Serialized 527 byte log entry
0112 02:52:40.019604 (+ 3592us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.019927 (+ 323us) write_op.cc:301] APPLY: starting
0112 02:52:40.019943 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.019983 (+ 40us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.020018 (+ 35us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.020061 (+ 43us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.020094 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:40.020103 (+ 9us) write_op.cc:312] APPLY: finished
0112 02:52:40.020169 (+ 66us) log.cc:844] Serialized 23 byte log entry
0112 02:52:40.020236 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.020253 (+ 17us) write_op.cc:454] Released schema lock
0112 02:52:40.020259 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:40.020334 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.020341 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":143,"reported_tablets":199,"child_traces":[["op",{"apply.queue_time_us":271,"apply.run_cpu_time_us":400,"apply.run_wall_time_us":399,"num_ops":1,"prepare.queue_time_us":81,"prepare.run_cpu_time_us":724,"prepare.run_wall_time_us":723,"raft.queue_time_us":80,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":247,"replication_time_us":3774}]]}
I20250112 02:52:41.482592 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet c29a9ca43e034b43a00207e5847bbee3 (table default.Table_9284616ca0cd499cb51df0322db9948f [id=7f36d8ba6d864d32ad84be06ca6682ee]) successfully deleted
I20250112 02:52:41.482774 2761 tablet.cc:1721] T 4d25815a5df94e099e7481c48c3da793 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:41.482633 2787 raft_consensus.cc:2743] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:41.483376 2696 leader_election.cc:290] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:41.483851 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec5346e13164274ae7bc3b6aa241f61" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:41.484632 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec5346e13164274ae7bc3b6aa241f61" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:41.487447 2762 tablet.cc:1721] T 4d25815a5df94e099e7481c48c3da793 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 02:52:41.484401 854 leader_election.cc:343] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:41.489492 855 leader_election.cc:343] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.490614 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 252) took 1253 ms. Trace:
I20250112 02:52:41.490819 2562 rpcz_store.cc:276] 0112 02:52:40.237412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.237454 (+ 42us) service_pool.cc:225] Handling call
0112 02:52:41.478295 (+1240841us) ts_tablet_manager.cc:1145] Deleting tablet bab4978edea0445fb0456e88e7fb4c44
0112 02:52:41.478304 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.490598 (+ 12294us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1240688}
I20250112 02:52:41.499904 2228 raft_consensus.cc:491] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.500037 2228 raft_consensus.cc:513] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.500669 2228 leader_election.cc:290] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:41.501199 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b7c2efd7ea402e8d19038f96b9675c" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:41.501329 1181 tablet_bootstrap.cc:654] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.501402 774 raft_consensus.cc:2463] T e4b7c2efd7ea402e8d19038f96b9675c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:41.501856 991 leader_election.cc:304] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:41.502136 2208 raft_consensus.cc:2798] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:41.502202 2208 raft_consensus.cc:491] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:41.502249 2208 raft_consensus.cc:3054] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.504552 2208 raft_consensus.cc:513] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.505061 2208 leader_election.cc:290] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:41.508273 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b7c2efd7ea402e8d19038f96b9675c" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:41.508440 774 raft_consensus.cc:3054] T e4b7c2efd7ea402e8d19038f96b9675c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.510843 774 raft_consensus.cc:2463] T e4b7c2efd7ea402e8d19038f96b9675c P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:41.511754 991 leader_election.cc:304] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:41.514504 2787 raft_consensus.cc:491] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.514629 2787 raft_consensus.cc:513] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.515198 2787 leader_election.cc:290] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:41.515970 2179 raft_consensus.cc:491] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.516113 2179 raft_consensus.cc:513] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.516744 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b7c2efd7ea402e8d19038f96b9675c" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:41.516826 2179 leader_election.cc:290] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:41.516906 911 raft_consensus.cc:3054] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.517163 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b7c2efd7ea402e8d19038f96b9675c" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:41.518344 855 leader_election.cc:304] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:41.518371 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b7c2efd7ea402e8d19038f96b9675c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:41.518548 1053 raft_consensus.cc:2388] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe36951a83fb4e57b26e187edbcbfe3d in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
I20250112 02:52:41.518625 2228 raft_consensus.cc:2798] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:41.518740 2228 raft_consensus.cc:695] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:41.518882 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9c845571f44c8eb8753090d63df45f" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:41.518951 2228 consensus_queue.cc:237] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.519170 2787 raft_consensus.cc:2743] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:41.522282 911 raft_consensus.cc:2463] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:41.522990 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b7c2efd7ea402e8d19038f96b9675c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:41.523200 774 raft_consensus.cc:2388] T e4b7c2efd7ea402e8d19038f96b9675c P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe36951a83fb4e57b26e187edbcbfe3d in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
I20250112 02:52:41.523610 855 leader_election.cc:304] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:41.523964 2787 raft_consensus.cc:2743] T e4b7c2efd7ea402e8d19038f96b9675c P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:41.524073 1173 consensus_queue.cc:260] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.524308 2326 raft_consensus.cc:491] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.524400 2326 raft_consensus.cc:513] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.525115 2326 leader_election.cc:290] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:41.525708 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2052b4536f9745f0a785f330d706c1cf" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:41.526293 991 leader_election.cc:343] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:41.526374 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet bab4978edea0445fb0456e88e7fb4c44 because the tablet was not found. No further retry: Not found: Tablet not found: bab4978edea0445fb0456e88e7fb4c44
I20250112 02:52:41.527282 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2052b4536f9745f0a785f330d706c1cf" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:41.527438 912 raft_consensus.cc:2463] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:41.527297 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9c845571f44c8eb8753090d63df45f" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:41.527774 916 raft_consensus.cc:2388] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 289afc116aa74408aa5bff74e11c50d8 in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
I20250112 02:52:41.527655 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_e87ea43667cd4f4eae136a1614aa9a67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:41.527992 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 253) took 1290 ms. Trace:
I20250112 02:52:41.528080 2562 rpcz_store.cc:276] 0112 02:52:40.237944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.237984 (+ 40us) service_pool.cc:225] Handling call
0112 02:52:41.491286 (+1253302us) ts_tablet_manager.cc:1145] Deleting tablet b1d9fae290ec4281839991f9d40b105c
0112 02:52:41.527209 (+ 35923us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.527979 (+ 770us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1253078}
W20250112 02:52:41.528515 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e87ea43667cd4f4eae136a1614aa9a67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:41.528585 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet b1d9fae290ec4281839991f9d40b105c because the tablet was not found. No further retry: Not found: Tablet not found: b1d9fae290ec4281839991f9d40b105c
I20250112 02:52:41.528676 713 leader_election.cc:304] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:41.528908 2451 raft_consensus.cc:3054] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.529399 989 leader_election.cc:304] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:41.529824 2326 raft_consensus.cc:2798] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:41.529896 2326 raft_consensus.cc:491] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:41.529953 2326 raft_consensus.cc:3054] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.530117 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_8d89d19f0fb24628b94e2c615bf9f787"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:41.530556 774 raft_consensus.cc:3054] T 46d1beac4410422bbf1e912d35999f3d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:41.530731 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d89d19f0fb24628b94e2c615bf9f787 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:41.532193 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_99896ca24c1a4eb1a31b036b021a8621"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:41.532388 2326 raft_consensus.cc:513] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:41.532734 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99896ca24c1a4eb1a31b036b021a8621 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:41.533525 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 254) took 1295 ms. Trace:
I20250112 02:52:41.534142 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_c80abe729cbf4e339b28b3ad6b3ba247"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:41.534643 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c80abe729cbf4e339b28b3ad6b3ba247 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:41.536206 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_8fe6c818f15341e08fe8870664eac4c1 [id=268dae304c9143a881c7a0898632a3e4] alter complete (version 2)
I20250112 02:52:41.537158 2451 raft_consensus.cc:2743] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:41.538033 2208 consensus_queue.cc:1035] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:41.545248 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2052b4536f9745f0a785f330d706c1cf" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
W20250112 02:52:41.545707 991 leader_election.cc:343] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.546892 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2052b4536f9745f0a785f330d706c1cf" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:41.547044 910 raft_consensus.cc:3054] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.548866 1031 tablet_service.cc:1467] Processing CreateTablet for tablet cca2a096fce0464394f5029652813873 (DEFAULT_TABLE table=default.Table_3185108f36ed429e9a9967cc2b11072e [id=a125c8e271a94d84a16b2ec6fb072674]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:41.549341 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 6b1d46c5395d4defb2d0e2948807eaf4 (DEFAULT_TABLE table=default.Table_8293882503214cf284e12ff8cc75b9c7 [id=15f8694f2e1a403fbf934065a1a79e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:41.549834 743 tablet_service.cc:1467] Processing CreateTablet for tablet cca2a096fce0464394f5029652813873 (DEFAULT_TABLE table=default.Table_3185108f36ed429e9a9967cc2b11072e [id=a125c8e271a94d84a16b2ec6fb072674]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:41.548381 2326 leader_election.cc:290] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:41.556071 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cca2a096fce0464394f5029652813873. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:41.556169 897 tablet_service.cc:1467] Processing CreateTablet for tablet cca2a096fce0464394f5029652813873 (DEFAULT_TABLE table=default.Table_3185108f36ed429e9a9967cc2b11072e [id=a125c8e271a94d84a16b2ec6fb072674]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:41.556366 895 tablet_service.cc:1467] Processing CreateTablet for tablet 6b1d46c5395d4defb2d0e2948807eaf4 (DEFAULT_TABLE table=default.Table_8293882503214cf284e12ff8cc75b9c7 [id=15f8694f2e1a403fbf934065a1a79e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:41.556481 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cca2a096fce0464394f5029652813873. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:41.556720 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b1d46c5395d4defb2d0e2948807eaf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:41.559094 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b1d46c5395d4defb2d0e2948807eaf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:41.559418 2179 raft_consensus.cc:491] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.559657 2179 raft_consensus.cc:513] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.559744 754 tablet_service.cc:1467] Processing CreateTablet for tablet 6b1d46c5395d4defb2d0e2948807eaf4 (DEFAULT_TABLE table=default.Table_8293882503214cf284e12ff8cc75b9c7 [id=15f8694f2e1a403fbf934065a1a79e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:41.560104 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b1d46c5395d4defb2d0e2948807eaf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:41.560256 2179 leader_election.cc:290] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:41.560618 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cca2a096fce0464394f5029652813873. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:41.561453 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1954e7859f422f9690fe4a9384a8bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.561803 760 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5b3c712c0f448ea9b14ab93dd731ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.562086 760 tablet_service.cc:1514] Processing DeleteTablet for tablet f924459d1150472aafb325a11825db23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:39 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.562366 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f4bd2df0ce45dabc55373f1154d3ac with delete_type TABLET_DATA_DELETED (replaced by tablet f559dedfb7854cf890bef362c50b4213) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.562620 760 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1b3ee229c445b19405cf4a19cf8003 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.562904 760 tablet_service.cc:1514] Processing DeleteTablet for tablet c29a9ca43e034b43a00207e5847bbee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.563154 760 tablet_service.cc:1514] Processing DeleteTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.563586 744 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ffdc237f8640f0b378809f93fc6292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.563657 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.563757 756 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d411f906b4d4eb68343bdcc493e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.563899 760 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e7882e07784bdda18dc1ffd20963c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.564189 760 tablet_service.cc:1514] Processing DeleteTablet for tablet b1d9fae290ec4281839991f9d40b105c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.564505 760 tablet_service.cc:1514] Processing DeleteTablet for tablet bab4978edea0445fb0456e88e7fb4c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:41.564781 713 leader_election.cc:343] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: bab4978edea0445fb0456e88e7fb4c44
I20250112 02:52:41.564819 760 tablet_service.cc:1514] Processing DeleteTablet for tablet f05c8426ea3d4fe89fbe6c88c688019f with delete_type TABLET_DATA_DELETED (replaced by tablet c5079f87652a4ea58b366d1f9448fd4e) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.565517 2003 tablet_replica.cc:331] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:41.565711 2003 raft_consensus.cc:2238] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:41.565833 2003 raft_consensus.cc:2267] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:41.566506 2003 ts_tablet_manager.cc:1905] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:41.574311 714 leader_election.cc:343] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: bab4978edea0445fb0456e88e7fb4c44
I20250112 02:52:41.579876 714 leader_election.cc:304] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
W20250112 02:52:41.580055 714 raft_consensus.cc:2681] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 02:52:41.580744 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c821ea3ca4946c290c17135366e0cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.581394 754 tablet_service.cc:1514] Processing DeleteTablet for tablet cbe30f9860e2492b90e8cedf8ff9daba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.582029 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ee98f055024429eafaa29bb961cab87 with delete_type TABLET_DATA_DELETED (replaced by tablet 99407ead9cf547e3bbe78c0d0b327e6a) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.582029 2745 consensus_queue.cc:1035] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:41.582952 2003 ts_tablet_manager.cc:1918] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:41.583056 2003 log.cc:1198] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/bab4978edea0445fb0456e88e7fb4c44
I20250112 02:52:41.583398 2003 ts_tablet_manager.cc:1939] T bab4978edea0445fb0456e88e7fb4c44 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:41.584549 2003 tablet_replica.cc:331] T 8c821ea3ca4946c290c17135366e0cac P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:41.584801 2003 raft_consensus.cc:2238] T 8c821ea3ca4946c290c17135366e0cac P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:41.584969 2003 raft_consensus.cc:2267] T 8c821ea3ca4946c290c17135366e0cac P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:41.585505 2003 ts_tablet_manager.cc:1905] T 8c821ea3ca4946c290c17135366e0cac P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:41.586462 910 raft_consensus.cc:2463] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:41.587479 966 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:41.587570 989 leader_election.cc:304] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:41.588086 2745 raft_consensus.cc:2798] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:41.588238 2745 raft_consensus.cc:695] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:41.588465 2745 consensus_queue.cc:237] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:41.592447 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 7e1954e7859f422f9690fe4a9384a8bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e1954e7859f422f9690fe4a9384a8bb already in progress: creating tablet
W20250112 02:52:41.592681 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet bd5b3c712c0f448ea9b14ab93dd731ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd5b3c712c0f448ea9b14ab93dd731ab already in progress: creating tablet
W20250112 02:52:41.593901 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f924459d1150472aafb325a11825db23 because the tablet was not found. No further retry: Not found: Tablet not found: f924459d1150472aafb325a11825db23
W20250112 02:52:41.594079 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 62f4bd2df0ce45dabc55373f1154d3ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62f4bd2df0ce45dabc55373f1154d3ac already in progress: creating tablet
W20250112 02:52:41.594642 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet ff1b3ee229c445b19405cf4a19cf8003 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff1b3ee229c445b19405cf4a19cf8003 already in progress: creating tablet
W20250112 02:52:41.594835 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet c29a9ca43e034b43a00207e5847bbee3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c29a9ca43e034b43a00207e5847bbee3 already in progress: creating tablet
I20250112 02:52:41.596326 1181 tablet_bootstrap.cc:492] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:41.596436 1181 ts_tablet_manager.cc:1397] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.225s user 0.008s sys 0.000s
I20250112 02:52:41.597029 1181 raft_consensus.cc:357] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.597195 1181 raft_consensus.cc:383] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.597239 1181 raft_consensus.cc:738] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.597453 1181 consensus_queue.cc:260] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.598459 1181 ts_tablet_manager.cc:1428] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:41.598558 2228 consensus_queue.cc:1035] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:41.598696 1181 tablet_bootstrap.cc:492] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
W20250112 02:52:41.523746 450 rpcz_store.cc:269] 0112 02:52:40.386221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.391940 (+ 5719us) service_pool.cc:225] Handling call
0112 02:52:41.394796 (+1002856us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":164}
I20250112 02:52:41.600988 2755 tablet.cc:1721] T 46d1beac4410422bbf1e912d35999f3d P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:52:41.606547 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f776e2a6ec704cf9b64483bc7d5b1dff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f776e2a6ec704cf9b64483bc7d5b1dff already in progress: creating tablet
W20250112 02:52:41.606752 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet a3ffdc237f8640f0b378809f93fc6292 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3ffdc237f8640f0b378809f93fc6292 already in progress: creating tablet
I20250112 02:52:41.606987 447 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 179) took 1025 ms. Trace:
I20250112 02:52:41.610823 2024 ts_tablet_manager.cc:1918] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250112 02:52:41.613153 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b because the tablet was not found. No further retry: Not found: Tablet not found: 4348fe4c32704bf7b3a8d21ff3b8ac1b
W20250112 02:52:41.613369 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d63d411f906b4d4eb68343bdcc493e05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d63d411f906b4d4eb68343bdcc493e05 already in progress: creating tablet
W20250112 02:52:41.613768 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet a8e7882e07784bdda18dc1ffd20963c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8e7882e07784bdda18dc1ffd20963c8 already in progress: creating tablet
W20250112 02:52:41.613973 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b1d9fae290ec4281839991f9d40b105c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1d9fae290ec4281839991f9d40b105c already in progress: creating tablet
I20250112 02:52:41.614859 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet bab4978edea0445fb0456e88e7fb4c44 (table default.Table_9dd5993a01b44a0abe33a5738c8d5c2b [id=6a7dcf7429b24699b036c53b3005051f]) successfully deleted
W20250112 02:52:41.615054 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f05c8426ea3d4fe89fbe6c88c688019f because the tablet was not found. No further retry: Not found: Tablet not found: f05c8426ea3d4fe89fbe6c88c688019f
I20250112 02:52:41.615278 1177 tablet_bootstrap.cc:492] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:41.615415 1177 ts_tablet_manager.cc:1397] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.206s user 0.002s sys 0.001s
I20250112 02:52:41.616756 754 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cb54003a83472cb9c57ace3e65b0be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.616851 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a34d03a80ae402db6607261a5c40b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.616968 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b351c563d749df806ab5d0331a4ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.617072 455 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 180) took 1034 ms. Trace:
I20250112 02:52:41.618918 454 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 181) took 1033 ms. Trace:
I20250112 02:52:41.618156 447 rpcz_store.cc:276] 0112 02:52:40.566742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.438664 (+871922us) service_pool.cc:225] Handling call
0112 02:52:41.532104 (+ 93440us) spinlock_profiling.cc:243] Waited 93.1 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd72d03 00007f3e13e75236 00007f3e13e74f65 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:41.532911 (+ 807us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:41.533366 (+ 455us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:41.581055 (+ 47689us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:41.592219 (+ 11164us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:41.592311 (+ 92us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:41.539095 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.539192 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.539198 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:41.539202 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:41.539281 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.539291 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.539298 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:41.539352 (+ 54us) tablet.cc:698] Row locks acquired
0112 02:52:41.539356 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:41.539384 (+ 28us) write_op.cc:270] Start()
0112 02:52:41.539413 (+ 29us) write_op.cc:276] Timestamp: P: 1736650361539295 usec, L: 0
0112 02:52:41.539418 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:41.539583 (+ 165us) log.cc:844] Serialized 433 byte log entry
0112 02:52:41.578002 (+ 38419us) op_driver.cc:464] REPLICATION: finished
0112 02:52:41.580032 (+ 2030us) write_op.cc:301] APPLY: starting
0112 02:52:41.580043 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:41.580077 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:41.580098 (+ 21us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:41.580196 (+ 98us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:41.580225 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:41.580231 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:41.580294 (+ 63us) log.cc:844] Serialized 29 byte log entry
0112 02:52:41.580356 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.580372 (+ 16us) write_op.cc:454] Released schema lock
0112 02:52:41.580377 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:41.580454 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.580458 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":214212352,"child_traces":[["op",{"apply.queue_time_us":1995,"apply.run_cpu_time_us":398,"apply.run_wall_time_us":398,"num_ops":2,"prepare.queue_time_us":5233,"prepare.run_cpu_time_us":550,"prepare.run_wall_time_us":550,"replication_time_us":38567}]]}
I20250112 02:52:41.622557 455 rpcz_store.cc:276] 0112 02:52:40.571174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.455267 (+884093us) service_pool.cc:225] Handling call
0112 02:52:41.530025 (+ 74758us) spinlock_profiling.cc:243] Waited 74.4 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd72d03 00007f3e13e75236 00007f3e13e74f65 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:41.530835 (+ 810us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:41.531369 (+ 534us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:41.591586 (+ 60217us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:41.605915 (+ 14329us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:41.606014 (+ 99us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:41.538433 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.538537 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.538542 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:41.538545 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:41.538632 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.538641 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.538645 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:41.538716 (+ 71us) tablet.cc:698] Row locks acquired
0112 02:52:41.538720 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:41.538750 (+ 30us) write_op.cc:270] Start()
0112 02:52:41.538775 (+ 25us) write_op.cc:276] Timestamp: P: 1736650361538661 usec, L: 0
0112 02:52:41.538778 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:41.538963 (+ 185us) log.cc:844] Serialized 433 byte log entry
0112 02:52:41.577641 (+ 38678us) op_driver.cc:464] REPLICATION: finished
0112 02:52:41.577892 (+ 251us) write_op.cc:301] APPLY: starting
0112 02:52:41.577909 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:41.577952 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:41.579521 (+ 1569us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:41.579642 (+ 121us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:41.579678 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:41.579684 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:41.579766 (+ 82us) log.cc:844] Serialized 29 byte log entry
0112 02:52:41.579833 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.579854 (+ 21us) write_op.cc:454] Released schema lock
0112 02:52:41.579875 (+ 21us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:41.579953 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.579960 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":171112704,"child_traces":[["op",{"apply.queue_time_us":192,"apply.run_cpu_time_us":528,"apply.run_wall_time_us":2051,"num_ops":2,"prepare.queue_time_us":6583,"prepare.run_cpu_time_us":606,"prepare.run_wall_time_us":606,"replication_time_us":38829}]]}
I20250112 02:52:41.628357 1177 raft_consensus.cc:357] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.629235 1177 raft_consensus.cc:383] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.630285 1177 raft_consensus.cc:738] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.630559 1177 consensus_queue.cc:260] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.634518 1177 ts_tablet_manager.cc:1428] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20250112 02:52:41.635115 2789 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:41.635757 2789 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:52:41.636722 2381 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: configured and running, proceeding with master startup.
W20250112 02:52:41.413702 452 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:38361 (request call id 214) took 1923 ms (client timeout 1000 ms). Trace:
I20250112 02:52:41.638211 1177 tablet_bootstrap.cc:492] T cf54f5185a0f4c70b9057309fca14e56 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:41.619036 454 rpcz_store.cc:276] 0112 02:52:40.577661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.457257 (+879596us) service_pool.cc:225] Handling call
0112 02:52:41.527499 (+ 70242us) spinlock_profiling.cc:243] Waited 68.5 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd72d03 00007f3e13e75236 00007f3e13e74f65 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:41.528699 (+ 1200us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:41.529223 (+ 524us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:41.610452 (+ 81229us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:41.610558 (+ 106us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:41.610633 (+ 75us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:41.536712 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.536843 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.536850 (+ 7us) write_op.cc:435] Acquired schema lock
0112 02:52:41.536866 (+ 16us) tablet.cc:659] Decoding operations
0112 02:52:41.536971 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.536985 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.536992 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:41.537085 (+ 93us) tablet.cc:698] Row locks acquired
0112 02:52:41.537090 (+ 5us) write_op.cc:260] PREPARE: finished
0112 02:52:41.537137 (+ 47us) write_op.cc:270] Start()
0112 02:52:41.537172 (+ 35us) write_op.cc:276] Timestamp: P: 1736650361537047 usec, L: 0
0112 02:52:41.537175 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:41.537327 (+ 152us) log.cc:844] Serialized 433 byte log entry
0112 02:52:41.542631 (+ 5304us) op_driver.cc:464] REPLICATION: finished
0112 02:52:41.543253 (+ 622us) write_op.cc:301] APPLY: starting
0112 02:52:41.543262 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:41.543290 (+ 28us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:41.543316 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:41.543397 (+ 81us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:41.543422 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:41.543428 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:41.543497 (+ 69us) log.cc:844] Serialized 29 byte log entry
0112 02:52:41.543552 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.543563 (+ 11us) write_op.cc:454] Released schema lock
0112 02:52:41.543568 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:41.543624 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.543627 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":157659904,"child_traces":[["op",{"apply.queue_time_us":589,"apply.run_cpu_time_us":364,"apply.run_wall_time_us":364,"num_ops":2,"prepare.queue_time_us":6918,"prepare.run_cpu_time_us":727,"prepare.run_wall_time_us":1666,"raft.queue_time_us":463,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":18,"replication_time_us":5436}]]}
I20250112 02:52:41.640933 1181 tablet_bootstrap.cc:654] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.641848 2766 tablet.cc:1721] T 4d25815a5df94e099e7481c48c3da793 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:41.642445 1177 tablet_bootstrap.cc:654] T cf54f5185a0f4c70b9057309fca14e56 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.646430 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 795ac7eaf81d48bb926865d9ebb388b6 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:41.629004 2003 ts_tablet_manager.cc:1918] T 8c821ea3ca4946c290c17135366e0cac P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:41.648834 2003 log.cc:1198] T 8c821ea3ca4946c290c17135366e0cac P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/8c821ea3ca4946c290c17135366e0cac
I20250112 02:52:41.649034 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b9280db29e54d41af1bed2fa5aa5535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.649264 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c04f1a2cb04950a9f6170e252c19b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.649525 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 54215c55966a46e1ab96e352c4e3764e with delete_type TABLET_DATA_DELETED (replaced by tablet a470f84d50df4cf0b4a1e5d248b4c32a) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.649818 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 70715db0394d46d9b6b289160b00be7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.650015 2283 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 3 FOLLOWER]: Advancing to term 4
I20250112 02:52:41.650381 754 tablet_service.cc:1514] Processing DeleteTablet for tablet e5c10fad790046a88fbc8ebe0303c481 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.650660 754 tablet_service.cc:1514] Processing DeleteTablet for tablet fe614eaeb793473d9b2da206b7345562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.650887 754 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a3486ef011492c8a1df8d6278adedf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.651223 754 tablet_service.cc:1514] Processing DeleteTablet for tablet f31b777e7e0f43a28839d8c48c6a6590 with delete_type TABLET_DATA_DELETED (replaced by tablet ca2165362ee541adac724b63f39b1489) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.651827 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 13578542e9e741e8817e0ee63b10f1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:41.655288 449 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:38361 (request call id 215) took 886 ms (client timeout 1000 ms). Trace:
I20250112 02:52:41.657397 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 87759e8b99b04e628badc8f60cb38bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.657589 454 master_service.cc:946] ReplaceTablet: received request to replace tablet b124c15736084e6d8f69b4fb7c41cf39 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:41.659984 2003 ts_tablet_manager.cc:1939] T 8c821ea3ca4946c290c17135366e0cac P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:41.660524 760 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d84027ad6d40a5be5afcdd854431a5 with delete_type TABLET_DATA_DELETED (replaced by tablet fbe9e9624ede40c29ea852b2b5494965) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.660634 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dda52f012794493a102ed1abc718f5b with delete_type TABLET_DATA_DELETED (replaced by tablet 17c54c5dc86446cf85974777c8145ef6) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.661410 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af55dab202444358ea1a0674d37946d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.661674 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 416fb6820cd54e118950c68d381f51b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:27 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.662119 451 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 182) took 1076 ms. Trace:
I20250112 02:52:41.662987 451 rpcz_store.cc:276] 0112 02:52:40.578253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.483942 (+905689us) service_pool.cc:225] Handling call
0112 02:52:41.534046 (+ 50104us) spinlock_profiling.cc:243] Waited 47.8 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd72d03 00007f3e13e75236 00007f3e13e74f65 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:41.534757 (+ 711us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:41.535171 (+ 414us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:41.642222 (+107051us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:41.654999 (+ 12777us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:41.655088 (+ 89us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:41.539685 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.539769 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.539775 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:41.539779 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:41.539896 (+ 117us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.539904 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.539908 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:41.539996 (+ 88us) tablet.cc:698] Row locks acquired
0112 02:52:41.539998 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:41.540022 (+ 24us) write_op.cc:270] Start()
0112 02:52:41.540046 (+ 24us) write_op.cc:276] Timestamp: P: 1736650361539935 usec, L: 0
0112 02:52:41.540051 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:41.540216 (+ 165us) log.cc:844] Serialized 433 byte log entry
0112 02:52:41.578040 (+ 37824us) op_driver.cc:464] REPLICATION: finished
0112 02:52:41.580516 (+ 2476us) write_op.cc:301] APPLY: starting
0112 02:52:41.580527 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:41.580554 (+ 27us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:41.580587 (+ 33us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:41.580666 (+ 79us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:41.580697 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:41.580703 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:41.580749 (+ 46us) log.cc:844] Serialized 29 byte log entry
0112 02:52:41.580796 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.580812 (+ 16us) write_op.cc:454] Released schema lock
0112 02:52:41.580818 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":109944064,"child_traces":[["op",{"apply.queue_time_us":2444,"apply.run_cpu_time_us":367,"apply.run_wall_time_us":354,"num_ops":2,"prepare.queue_time_us":4146,"prepare.run_cpu_time_us":586,"prepare.run_wall_time_us":585,"replication_time_us":37976,"threads_started":1}]]}
I20250112 02:52:41.667085 2024 log.cc:1198] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/fe614eaeb793473d9b2da206b7345562
W20250112 02:52:41.662242 449 rpcz_store.cc:269] 0112 02:52:40.768390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.863283 (+ 94893us) service_pool.cc:225] Handling call
0112 02:52:41.161618 (+298335us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.161624 (+ 6us) write_op.cc:454] Released schema lock
0112 02:52:41.655262 (+493638us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:41.154669 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.154761 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.154767 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:41.154769 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:41.154904 (+ 135us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.154915 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.154919 (+ 4us) tablet.cc:682] Acquiring locks for 5 operations
0112 02:52:41.155045 (+ 126us) tablet.cc:698] Row locks acquired
0112 02:52:41.155047 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:41.155088 (+ 41us) write_op.cc:270] Start()
0112 02:52:41.155116 (+ 28us) write_op.cc:276] Timestamp: P: 1736650361154977 usec, L: 0
0112 02:52:41.155119 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:41.155282 (+ 163us) log.cc:844] Serialized 2024 byte log entry
0112 02:52:41.158223 (+ 2941us) op_driver.cc:464] REPLICATION: finished
0112 02:52:41.158308 (+ 85us) write_op.cc:301] APPLY: starting
0112 02:52:41.158321 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:41.158398 (+ 77us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:41.158421 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:41.158507 (+ 86us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:41.158541 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0112 02:52:41.158547 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:41.158616 (+ 69us) log.cc:844] Serialized 47 byte log entry
0112 02:52:41.158680 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.158699 (+ 19us) write_op.cc:454] Released schema lock
0112 02:52:41.158705 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":22130,"reported_tablets":178,"child_traces":[["op",{"apply.queue_time_us":47,"apply.run_cpu_time_us":443,"apply.run_wall_time_us":443,"num_ops":5,"prepare.queue_time_us":1310,"prepare.run_cpu_time_us":708,"prepare.run_wall_time_us":708,"raft.queue_time_us":268,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":218,"replication_time_us":3078,"spinlock_wait_cycles":32000}]]}
I20250112 02:52:41.628487 1173 ts_tablet_manager.cc:1428] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.220s user 0.002s sys 0.000s
I20250112 02:52:41.668345 1173 tablet_bootstrap.cc:492] T b1d9fae290ec4281839991f9d40b105c P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:41.669667 1173 tablet_bootstrap.cc:654] T b1d9fae290ec4281839991f9d40b105c P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.672080 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_6e552be8f8b7468f84f7675b3342125e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:41.672619 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e552be8f8b7468f84f7675b3342125e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:41.674587 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_80b349ce76644bcabf626e044231436a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:41.675308 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80b349ce76644bcabf626e044231436a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:41.676316 2024 ts_tablet_manager.cc:1939] T fe614eaeb793473d9b2da206b7345562 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:41.676704 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 231) took 2683 ms. Trace:
I20250112 02:52:41.676846 2024 rpcz_store.cc:276] 0112 02:52:38.993247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.993424 (+ 177us) service_pool.cc:225] Handling call
0112 02:52:41.395325 (+2401901us) ts_tablet_manager.cc:1145] Deleting tablet fe614eaeb793473d9b2da206b7345562
0112 02:52:41.395335 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.607891 (+212556us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.610750 (+ 2859us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.676688 (+ 65938us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2401770}
I20250112 02:52:41.677284 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 226) took 2685 ms. Trace:
I20250112 02:52:41.677398 2024 rpcz_store.cc:276] 0112 02:52:38.991982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:38.993587 (+ 1605us) service_pool.cc:225] Handling call
0112 02:52:41.677182 (+2683595us) ts_tablet_manager.cc:1145] Deleting tablet 4c0c2fad4f3243238641aae56b305bb5
0112 02:52:41.677190 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.677270 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2683462}
I20250112 02:52:41.677894 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 233) took 2676 ms. Trace:
I20250112 02:52:41.678020 2024 rpcz_store.cc:276] 0112 02:52:39.001180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.001223 (+ 43us) service_pool.cc:225] Handling call
0112 02:52:41.677808 (+2676585us) ts_tablet_manager.cc:1145] Deleting tablet 35f59617b34046568dd83bbb7cdc42d2
0112 02:52:41.677816 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.677880 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2676409}
I20250112 02:52:41.678345 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 232) took 2677 ms. Trace:
I20250112 02:52:41.678457 2024 rpcz_store.cc:276] 0112 02:52:39.001083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.001320 (+ 237us) service_pool.cc:225] Handling call
0112 02:52:41.678273 (+2676953us) ts_tablet_manager.cc:1145] Deleting tablet b2d8ea2ef18a46508371b4789ab6f4fa
0112 02:52:41.678280 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.678332 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2676788}
I20250112 02:52:41.678755 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 234) took 2677 ms. Trace:
I20250112 02:52:41.679942 756 tablet_service.cc:1514] Processing DeleteTablet for tablet f3771166b77e4f90aaa116ab1a9dc299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:39 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.680330 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 058a9ca089cd4967887b0a1127c7d03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.680945 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 06edfca60eed4dd3a2dee7b662b0970c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.681674 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 8c821ea3ca4946c290c17135366e0cac (table default.Table_0f1762619e9543018fd2df57adb6367e [id=5f5a61b2a00d4d29bd79a696474a787b]) successfully deleted
W20250112 02:52:41.681955 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet cbe30f9860e2492b90e8cedf8ff9daba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbe30f9860e2492b90e8cedf8ff9daba already in progress: creating tablet
W20250112 02:52:41.656865 452 rpcz_store.cc:269] 0112 02:52:39.490500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.153843 (+663343us) service_pool.cc:225] Handling call
0112 02:52:41.413662 (+1259819us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:40.640825 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.640922 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.640928 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:40.640930 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:40.641086 (+ 156us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.641099 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.641103 (+ 4us) tablet.cc:682] Acquiring locks for 3 operations
0112 02:52:40.641184 (+ 81us) tablet.cc:698] Row locks acquired
0112 02:52:40.641186 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:40.641218 (+ 32us) write_op.cc:270] Start()
0112 02:52:40.641246 (+ 28us) write_op.cc:276] Timestamp: P: 1736650360641106 usec, L: 0
0112 02:52:40.641249 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.641427 (+ 178us) log.cc:844] Serialized 1275 byte log entry
0112 02:52:40.644628 (+ 3201us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.644759 (+ 131us) write_op.cc:301] APPLY: starting
0112 02:52:40.644771 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.644841 (+ 70us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.644899 (+ 58us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.645031 (+ 132us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.645076 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0112 02:52:40.645085 (+ 9us) write_op.cc:312] APPLY: finished
0112 02:52:40.645197 (+ 112us) log.cc:844] Serialized 35 byte log entry
0112 02:52:40.645253 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.645270 (+ 17us) write_op.cc:454] Released schema lock
0112 02:52:40.645276 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:40.645342 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.645345 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":160920,"reported_tablets":186,"child_traces":[["op",{"apply.queue_time_us":70,"apply.run_cpu_time_us":578,"apply.run_wall_time_us":578,"num_ops":3,"prepare.queue_time_us":285,"prepare.run_cpu_time_us":703,"prepare.run_wall_time_us":703,"raft.queue_time_us":471,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":237,"replication_time_us":3353}]]}
I20250112 02:52:41.683168 1181 tablet_bootstrap.cc:492] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:41.683336 1181 ts_tablet_manager.cc:1397] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.001s
I20250112 02:52:41.684206 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 003a3b4705e943c6a900762ec11ddd47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.684108 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_ea6f20bc46ec487cae9e3abf6419c80c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:41.684582 1181 raft_consensus.cc:357] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:41.684768 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea6f20bc46ec487cae9e3abf6419c80c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:41.684901 1181 raft_consensus.cc:383] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.685019 1181 raft_consensus.cc:738] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.685333 1181 consensus_queue.cc:260] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.686044 1181 ts_tablet_manager.cc:1428] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:41.686331 1181 tablet_bootstrap.cc:492] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:41.687731 1181 tablet_bootstrap.cc:654] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:41.690831 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 8ee98f055024429eafaa29bb961cab87 because the tablet was not found. No further retry: Not found: Tablet not found: 8ee98f055024429eafaa29bb961cab87
W20250112 02:52:41.691161 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d0cb54003a83472cb9c57ace3e65b0be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0cb54003a83472cb9c57ace3e65b0be already in progress: creating tablet
W20250112 02:52:41.691417 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 7a34d03a80ae402db6607261a5c40b60 because the tablet was not found. No further retry: Not found: Tablet not found: 7a34d03a80ae402db6607261a5c40b60
W20250112 02:52:41.691711 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 82b351c563d749df806ab5d0331a4ef2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82b351c563d749df806ab5d0331a4ef2 already in progress: creating tablet
W20250112 02:52:41.691972 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 3b9280db29e54d41af1bed2fa5aa5535 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b9280db29e54d41af1bed2fa5aa5535 already in progress: creating tablet
W20250112 02:52:41.692250 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 83c04f1a2cb04950a9f6170e252c19b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83c04f1a2cb04950a9f6170e252c19b4 already in progress: creating tablet
W20250112 02:52:41.692516 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 54215c55966a46e1ab96e352c4e3764e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54215c55966a46e1ab96e352c4e3764e already in progress: creating tablet
W20250112 02:52:41.692776 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 70715db0394d46d9b6b289160b00be7d because the tablet was not found. No further retry: Not found: Tablet not found: 70715db0394d46d9b6b289160b00be7d
I20250112 02:52:41.693010 1177 tablet_bootstrap.cc:492] T cf54f5185a0f4c70b9057309fca14e56 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
W20250112 02:52:41.693018 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet e5c10fad790046a88fbc8ebe0303c481 because the tablet was not found. No further retry: Not found: Tablet not found: e5c10fad790046a88fbc8ebe0303c481
I20250112 02:52:41.693173 1177 ts_tablet_manager.cc:1397] T cf54f5185a0f4c70b9057309fca14e56 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.004s
W20250112 02:52:41.693305 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet fe614eaeb793473d9b2da206b7345562 because the tablet was not found. No further retry: Not found: Tablet not found: fe614eaeb793473d9b2da206b7345562
W20250112 02:52:41.693537 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet c4a3486ef011492c8a1df8d6278adedf because the tablet was not found. No further retry: Not found: Tablet not found: c4a3486ef011492c8a1df8d6278adedf
W20250112 02:52:41.693766 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f31b777e7e0f43a28839d8c48c6a6590 because the tablet was not found. No further retry: Not found: Tablet not found: f31b777e7e0f43a28839d8c48c6a6590
W20250112 02:52:41.694012 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 13578542e9e741e8817e0ee63b10f1a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13578542e9e741e8817e0ee63b10f1a1 already in progress: creating tablet
I20250112 02:52:41.693944 1177 raft_consensus.cc:357] T cf54f5185a0f4c70b9057309fca14e56 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.694154 1177 raft_consensus.cc:383] T cf54f5185a0f4c70b9057309fca14e56 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.694217 1177 raft_consensus.cc:738] T cf54f5185a0f4c70b9057309fca14e56 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
W20250112 02:52:41.694366 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b6d84027ad6d40a5be5afcdd854431a5 because the tablet was not found. No further retry: Not found: Tablet not found: b6d84027ad6d40a5be5afcdd854431a5
I20250112 02:52:41.694516 1177 consensus_queue.cc:260] T cf54f5185a0f4c70b9057309fca14e56 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:41.694610 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 1dda52f012794493a102ed1abc718f5b because the tablet was not found. No further retry: Not found: Tablet not found: 1dda52f012794493a102ed1abc718f5b
W20250112 02:52:41.694898 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 9af55dab202444358ea1a0674d37946d because the tablet was not found. No further retry: Not found: Tablet not found: 9af55dab202444358ea1a0674d37946d
I20250112 02:52:41.695154 1177 ts_tablet_manager.cc:1428] T cf54f5185a0f4c70b9057309fca14e56 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250112 02:52:41.695150 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 416fb6820cd54e118950c68d381f51b2 because the tablet was not found. No further retry: Not found: Tablet not found: 416fb6820cd54e118950c68d381f51b2
W20250112 02:52:41.695385 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 87759e8b99b04e628badc8f60cb38bb3 because the tablet was not found. No further retry: Not found: Tablet not found: 87759e8b99b04e628badc8f60cb38bb3
W20250112 02:52:41.695636 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f3771166b77e4f90aaa116ab1a9dc299 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3771166b77e4f90aaa116ab1a9dc299 already in progress: creating tablet
W20250112 02:52:41.695844 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 058a9ca089cd4967887b0a1127c7d03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 058a9ca089cd4967887b0a1127c7d03f already in progress: creating tablet
W20250112 02:52:41.696054 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 06edfca60eed4dd3a2dee7b662b0970c because the tablet was not found. No further retry: Not found: Tablet not found: 06edfca60eed4dd3a2dee7b662b0970c
W20250112 02:52:41.696307 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 003a3b4705e943c6a900762ec11ddd47 because the tablet was not found. No further retry: Not found: Tablet not found: 003a3b4705e943c6a900762ec11ddd47
W20250112 02:52:41.633926 989 consensus_peers.cc:487] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f924459d1150472aafb325a11825db23. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:41.629094 1102 heartbeater.cc:346] Connected to a master server at 127.0.104.61:38771
I20250112 02:52:41.691785 2745 consensus_queue.cc:1035] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:41.697043 1177 tablet_bootstrap.cc:492] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:41.697063 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 63169.4us
I20250112 02:52:41.697522 743 tablet_service.cc:1514] Processing DeleteTablet for tablet c29a9ca43e034b43a00207e5847bbee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.697706 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet fe614eaeb793473d9b2da206b7345562 (table default.Table_b2b3c738a24f451eac39fb1a6399f37a [id=7c6c2713869e476cb23d157a285a5afd]) successfully deleted
W20250112 02:52:41.697966 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 4c0c2fad4f3243238641aae56b305bb5 because the tablet was not found. No further retry: Not found: Tablet not found: 4c0c2fad4f3243238641aae56b305bb5
W20250112 02:52:41.698168 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 35f59617b34046568dd83bbb7cdc42d2 because the tablet was not found. No further retry: Not found: Tablet not found: 35f59617b34046568dd83bbb7cdc42d2
I20250112 02:52:41.698410 1177 tablet_bootstrap.cc:654] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:41.698407 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b2d8ea2ef18a46508371b4789ab6f4fa because the tablet was not found. No further retry: Not found: Tablet not found: b2d8ea2ef18a46508371b4789ab6f4fa
W20250112 02:52:41.698601 453 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:37987 (request call id 256) took 1282 ms (client timeout 1000 ms). Trace:
I20250112 02:52:41.699049 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 with delete_type TABLET_DATA_DELETED (replaced by tablet 3dab0466476c4a9aa265271c1abd91a4) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:41.698693 453 rpcz_store.cc:269] 0112 02:52:40.415821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.459176 (+ 43355us) service_pool.cc:225] Handling call
0112 02:52:40.940306 (+481130us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.940311 (+ 5us) write_op.cc:454] Released schema lock
0112 02:52:41.698550 (+758239us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:40.935427 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.935541 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.935546 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:40.935548 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:40.935673 (+ 125us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.935684 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.935687 (+ 3us) tablet.cc:682] Acquiring locks for 3 operations
0112 02:52:40.935756 (+ 69us) tablet.cc:698] Row locks acquired
0112 02:52:40.935759 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:40.935810 (+ 51us) write_op.cc:270] Start()
0112 02:52:40.935832 (+ 22us) write_op.cc:276] Timestamp: P: 1736650360935742 usec, L: 0
0112 02:52:40.935836 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.935985 (+ 149us) log.cc:844] Serialized 1275 byte log entry
0112 02:52:40.938116 (+ 2131us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.938227 (+ 111us) write_op.cc:301] APPLY: starting
0112 02:52:40.938239 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.938277 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.938299 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.938360 (+ 61us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.938386 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0112 02:52:40.938393 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:40.938470 (+ 77us) log.cc:844] Serialized 35 byte log entry
0112 02:52:40.938523 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.938539 (+ 16us) write_op.cc:454] Released schema lock
0112 02:52:40.938545 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":142889,"reported_tablets":175,"child_traces":[["op",{"apply.queue_time_us":58,"apply.run_cpu_time_us":362,"apply.run_wall_time_us":361,"num_ops":3,"prepare.queue_time_us":522,"prepare.run_cpu_time_us":645,"prepare.run_wall_time_us":644,"raft.queue_time_us":50,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":184,"replication_time_us":2256}]]}
I20250112 02:52:41.701090 2024 rpcz_store.cc:276] 0112 02:52:39.001277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.001425 (+ 148us) service_pool.cc:225] Handling call
0112 02:52:41.678695 (+2677270us) ts_tablet_manager.cc:1145] Deleting tablet 42cb7e7a41634cf1bdb978ac9d2befcd
0112 02:52:41.678702 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.678745 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2677096}
W20250112 02:52:41.701541 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 42cb7e7a41634cf1bdb978ac9d2befcd because the tablet was not found. No further retry: Not found: Tablet not found: 42cb7e7a41634cf1bdb978ac9d2befcd
W20250112 02:52:41.702957 448 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:37987 (request call id 255) took 2367 ms (client timeout 1000 ms). Trace:
I20250112 02:52:41.703588 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 236) took 2699 ms. Trace:
I20250112 02:52:41.703708 2024 rpcz_store.cc:276] 0112 02:52:39.004288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.004330 (+ 42us) service_pool.cc:225] Handling call
0112 02:52:41.702121 (+2697791us) ts_tablet_manager.cc:1145] Deleting tablet ac716c6da01941a989754aac1ef430d5
0112 02:52:41.703501 (+ 1380us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.703570 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2697607}
W20250112 02:52:41.704250 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet ac716c6da01941a989754aac1ef430d5 because the tablet was not found. No further retry: Not found: Tablet not found: ac716c6da01941a989754aac1ef430d5
I20250112 02:52:41.704285 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 238) took 2699 ms. Trace:
I20250112 02:52:41.704349 2024 rpcz_store.cc:276] 0112 02:52:39.004450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.004493 (+ 43us) service_pool.cc:225] Handling call
0112 02:52:41.704217 (+2699724us) ts_tablet_manager.cc:1145] Deleting tablet f6803d0f553b4204b415b2734286bf4a
0112 02:52:41.704226 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.704271 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2699542}
W20250112 02:52:41.704653 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f6803d0f553b4204b415b2734286bf4a because the tablet was not found. No further retry: Not found: Tablet not found: f6803d0f553b4204b415b2734286bf4a
I20250112 02:52:41.705245 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_7926c082d6c04315811ea3b4e1579e8c" } new_table_name: "default.Table_f952a0479e8d478bbd9484fc5bf041c8" modify_external_catalogs: true
W20250112 02:52:41.709059 448 rpcz_store.cc:269] 0112 02:52:39.332975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.826382 (+493407us) service_pool.cc:225] Handling call
0112 02:52:41.699979 (+1873597us) spinlock_profiling.cc:243] Waited 9.32 ms on lock 0x60f00002f058. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 000000000096ceda 00007f3e098520fd 00007f3e09851d1e 00007f3e1cc38f3d 00007f3e1cc38edc 00007f3e0aa2e813 00007f3e1cd700eb 00007f3e13e72716 00007f3e13e72445 00007f3e105d130e 00007f3e105d01ee
0112 02:52:41.700199 (+ 220us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:40.178797 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.178908 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.178913 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:40.178915 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:40.179018 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.179029 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.179033 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:40.179117 (+ 84us) tablet.cc:698] Row locks acquired
0112 02:52:40.179121 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:40.179154 (+ 33us) write_op.cc:270] Start()
0112 02:52:40.179182 (+ 28us) write_op.cc:276] Timestamp: P: 1736650360179134 usec, L: 0
0112 02:52:40.179185 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.179339 (+ 154us) log.cc:844] Serialized 901 byte log entry
0112 02:52:40.188655 (+ 9316us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.188756 (+ 101us) write_op.cc:301] APPLY: starting
0112 02:52:40.188768 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.188814 (+ 46us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.188858 (+ 44us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.188926 (+ 68us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.188982 (+ 56us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:40.188993 (+ 11us) write_op.cc:312] APPLY: finished
0112 02:52:40.189071 (+ 78us) log.cc:844] Serialized 29 byte log entry
0112 02:52:40.189136 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.189150 (+ 14us) write_op.cc:454] Released schema lock
0112 02:52:40.189155 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:40.189214 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.189219 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:40.197702 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:40.197788 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:40.197795 (+ 7us) write_op.cc:435] Acquired schema lock
0112 02:52:40.197798 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:40.197876 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:40.197885 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:40.197892 (+ 7us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:40.197939 (+ 47us) tablet.cc:698] Row locks acquired
0112 02:52:40.197943 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:40.197977 (+ 34us) write_op.cc:270] Start()
0112 02:52:40.198001 (+ 24us) write_op.cc:276] Timestamp: P: 1736650360197954 usec, L: 0
0112 02:52:40.198006 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:40.198137 (+ 131us) log.cc:844] Serialized 356 byte log entry
0112 02:52:40.337374 (+139237us) op_driver.cc:464] REPLICATION: finished
0112 02:52:40.337438 (+ 64us) write_op.cc:301] APPLY: starting
0112 02:52:40.337448 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:40.337469 (+ 21us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:40.337487 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:40.337521 (+ 34us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:40.337539 (+ 18us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:40.337545 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:40.337588 (+ 43us) log.cc:844] Serialized 23 byte log entry
0112 02:52:40.337625 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:40.337637 (+ 12us) write_op.cc:454] Released schema lock
0112 02:52:40.337642 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 02:52:41.410269 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.410379 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.410387 (+ 8us) write_op.cc:435] Acquired schema lock
0112 02:52:41.410390 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:41.410489 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.410500 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.410506 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:41.410562 (+ 56us) tablet.cc:698] Row locks acquired
0112 02:52:41.410566 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:41.410597 (+ 31us) write_op.cc:270] Start()
0112 02:52:41.410620 (+ 23us) write_op.cc:276] Timestamp: P: 1736650361410533 usec, L: 0
0112 02:52:41.410623 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:41.410754 (+ 131us) log.cc:844] Serialized 356 byte log entry
0112 02:52:41.526926 (+116172us) op_driver.cc:464] REPLICATION: finished
0112 02:52:41.535591 (+ 8665us) write_op.cc:301] APPLY: starting
0112 02:52:41.535608 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:41.535649 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:41.535709 (+ 60us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:41.535765 (+ 56us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:41.535798 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:41.535805 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:41.535930 (+ 125us) log.cc:844] Serialized 23 byte log entry
0112 02:52:41.536009 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.536030 (+ 21us) write_op.cc:454] Released schema lock
0112 02:52:41.536038 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:41.536113 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.536117 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":190,"reported_tablets":187,"spinlock_wait_cycles":21435648,"child_traces":[["op",{"apply.queue_time_us":61,"apply.run_cpu_time_us":450,"apply.run_wall_time_us":449,"num_ops":2,"prepare.queue_time_us":2628,"prepare.run_cpu_time_us":633,"prepare.run_wall_time_us":633,"raft.queue_time_us":434,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":213,"replication_time_us":9445}],["op",{"apply.queue_time_us":29,"apply.run_cpu_time_us":245,"apply.run_wall_time_us":244,"num_ops":1,"prepare.queue_time_us":1291,"prepare.run_cpu_time_us":469,"prepare.run_wall_time_us":469,"replication_time_us":139348}],["op",{"apply.queue_time_us":8586,"apply.run_cpu_time_us":519,"apply.run_wall_time_us":518,"mutex_wait_us":3209,"num_ops":1,"prepare.queue_time_us":3795,"prepare.run_cpu_time_us":548,"prepare.run_wall_time_us":548,"replication_time_us":116264}]]}
I20250112 02:52:41.710821 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_104ccd5f390f47a5878c67ea6de9d554" } modify_external_catalogs: true
I20250112 02:52:41.710999 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_104ccd5f390f47a5878c67ea6de9d554" } modify_external_catalogs: true
I20250112 02:52:41.711897 1173 tablet_bootstrap.cc:492] T b1d9fae290ec4281839991f9d40b105c P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:41.712038 1173 ts_tablet_manager.cc:1397] T b1d9fae290ec4281839991f9d40b105c P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20250112 02:52:41.713289 2283 pending_rounds.cc:77] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Aborting all ops after (but not including) 514
I20250112 02:52:41.718129 2609 consensus_queue.cc:1035] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:41.719794 754 tablet_service.cc:1514] Processing DeleteTablet for tablet bab4978edea0445fb0456e88e7fb4c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.700922 2562 rpcz_store.cc:276] 0112 02:52:40.238240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.238273 (+ 33us) service_pool.cc:225] Handling call
0112 02:52:41.528343 (+1290070us) ts_tablet_manager.cc:1145] Deleting tablet a8e7882e07784bdda18dc1ffd20963c8
0112 02:52:41.533386 (+ 5043us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.533490 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1289964}
I20250112 02:52:41.721861 2562 tablet_replica.cc:331] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:41.722023 2562 raft_consensus.cc:2238] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:41.722386 2562 raft_consensus.cc:2267] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:41.722837 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet a8e7882e07784bdda18dc1ffd20963c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8e7882e07784bdda18dc1ffd20963c8 already in progress: creating tablet
I20250112 02:52:41.722820 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 239) took 2718 ms. Trace:
I20250112 02:52:41.722968 2024 rpcz_store.cc:276] 0112 02:52:39.004543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.004580 (+ 37us) service_pool.cc:225] Handling call
0112 02:52:41.712366 (+2707786us) ts_tablet_manager.cc:1145] Deleting tablet cc62dcbf67e84e36ae1fbe127ccd4879
0112 02:52:41.712377 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.722797 (+ 10420us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2707559}
I20250112 02:52:41.723816 2562 ts_tablet_manager.cc:1905] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:41.723897 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet cc62dcbf67e84e36ae1fbe127ccd4879 because the tablet was not found. No further retry: Not found: Tablet not found: cc62dcbf67e84e36ae1fbe127ccd4879
I20250112 02:52:41.725214 2283 pending_rounds.cc:99] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Aborting uncommitted WRITE_OP operation due to leader change: 3.515
I20250112 02:52:41.725854 1177 tablet_bootstrap.cc:492] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:41.725950 1177 ts_tablet_manager.cc:1397] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.005s
I20250112 02:52:41.726255 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 240) took 2721 ms. Trace:
I20250112 02:52:41.726320 2024 rpcz_store.cc:276] 0112 02:52:39.004656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.004898 (+ 242us) service_pool.cc:225] Handling call
0112 02:52:41.723401 (+2718503us) ts_tablet_manager.cc:1145] Deleting tablet 3c686fdc8896463c81c9f7c5f53a3da2
0112 02:52:41.723412 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.726238 (+ 2826us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2718316}
W20250112 02:52:41.726683 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 3c686fdc8896463c81c9f7c5f53a3da2 because the tablet was not found. No further retry: Not found: Tablet not found: 3c686fdc8896463c81c9f7c5f53a3da2
I20250112 02:52:41.727376 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 242) took 2722 ms. Trace:
I20250112 02:52:41.727320 1177 raft_consensus.cc:357] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.727504 1177 raft_consensus.cc:383] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.727450 2024 rpcz_store.cc:276] 0112 02:52:39.004858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.004957 (+ 99us) service_pool.cc:225] Handling call
0112 02:52:41.726525 (+2721568us) ts_tablet_manager.cc:1145] Deleting tablet 6b521522c4784d2ab4056268823f1099
0112 02:52:41.726533 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.727347 (+ 814us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2721377}
I20250112 02:52:41.727566 1177 raft_consensus.cc:738] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
W20250112 02:52:41.727916 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 6b521522c4784d2ab4056268823f1099 because the tablet was not found. No further retry: Not found: Tablet not found: 6b521522c4784d2ab4056268823f1099
I20250112 02:52:41.727828 1173 raft_consensus.cc:357] T b1d9fae290ec4281839991f9d40b105c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.727769 1177 consensus_queue.cc:260] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.728026 1173 raft_consensus.cc:383] T b1d9fae290ec4281839991f9d40b105c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.728076 1173 raft_consensus.cc:738] T b1d9fae290ec4281839991f9d40b105c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.728240 1173 consensus_queue.cc:260] T b1d9fae290ec4281839991f9d40b105c P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.728451 1177 ts_tablet_manager.cc:1428] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 02:52:41.728667 1177 tablet_bootstrap.cc:492] T fea2ef4e9a63491db850bb5393938017 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:41.728770 1173 ts_tablet_manager.cc:1428] T b1d9fae290ec4281839991f9d40b105c P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250112 02:52:41.729784 1177 tablet_bootstrap.cc:654] T fea2ef4e9a63491db850bb5393938017 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.733657 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 235) took 2729 ms. Trace:
I20250112 02:52:41.733726 2024 rpcz_store.cc:276] 0112 02:52:39.004198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.005168 (+ 970us) service_pool.cc:225] Handling call
0112 02:52:41.727794 (+2722626us) ts_tablet_manager.cc:1145] Deleting tablet 7f4e4f1e451b48dc9cdd780300f6888c
0112 02:52:41.733567 (+ 5773us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.733644 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2722461}
I20250112 02:52:41.734395 2024 tablet_replica.cc:331] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:41.734591 2024 raft_consensus.cc:2238] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
W20250112 02:52:41.734591 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 7f4e4f1e451b48dc9cdd780300f6888c because the tablet was not found. No further retry: Not found: Tablet not found: 7f4e4f1e451b48dc9cdd780300f6888c
I20250112 02:52:41.734887 2024 pending_rounds.cc:62] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6: Trying to abort 1 pending ops.
I20250112 02:52:41.735031 2024 pending_rounds.cc:69] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319871222550528 op_type: NO_OP noop_request { }
I20250112 02:52:41.735240 2024 raft_consensus.cc:2883] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:41.735347 2283 pending_rounds.cc:99] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Aborting uncommitted WRITE_OP operation due to leader change: 3.516
I20250112 02:52:41.735539 1181 tablet_bootstrap.cc:492] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:41.735674 1181 ts_tablet_manager.cc:1397] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20250112 02:52:41.735705 2024 raft_consensus.cc:2267] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:41.736951 2024 ts_tablet_manager.cc:1905] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:41.737241 1181 raft_consensus.cc:357] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.737910 2283 pending_rounds.cc:99] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Aborting uncommitted WRITE_OP operation due to leader change: 3.517
I20250112 02:52:41.738381 2283 pending_rounds.cc:99] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Aborting uncommitted WRITE_OP operation due to leader change: 3.518
I20250112 02:52:41.738492 1181 raft_consensus.cc:383] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.736409 1173 tablet_bootstrap.cc:492] T 6171b8ff2bfe41d598b0cb2535799a11 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:41.741654 1173 tablet_bootstrap.cc:654] T 6171b8ff2bfe41d598b0cb2535799a11 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.742341 1181 raft_consensus.cc:738] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.742616 1181 consensus_queue.cc:260] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.743026 2283 pending_rounds.cc:99] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Aborting uncommitted WRITE_OP operation due to leader change: 3.519
I20250112 02:52:41.743542 1181 ts_tablet_manager.cc:1428] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250112 02:52:41.743808 2283 pending_rounds.cc:99] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Aborting uncommitted WRITE_OP operation due to leader change: 3.520
I20250112 02:52:41.744060 2283 pending_rounds.cc:99] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Aborting uncommitted WRITE_OP operation due to leader change: 3.521
I20250112 02:52:41.744325 1181 tablet_bootstrap.cc:492] T 54d3664071b146a3ac7c6885f9bda789 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:41.744529 1173 tablet_bootstrap.cc:492] T 6171b8ff2bfe41d598b0cb2535799a11 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:41.744642 1173 ts_tablet_manager.cc:1397] T 6171b8ff2bfe41d598b0cb2535799a11 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250112 02:52:41.745287 1173 raft_consensus.cc:357] T 6171b8ff2bfe41d598b0cb2535799a11 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.745463 1173 raft_consensus.cc:383] T 6171b8ff2bfe41d598b0cb2535799a11 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.744947 2283 pending_rounds.cc:99] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: Aborting uncommitted WRITE_OP operation due to leader change: 3.522
I20250112 02:52:41.745522 1173 raft_consensus.cc:738] T 6171b8ff2bfe41d598b0cb2535799a11 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.745682 1173 consensus_queue.cc:260] T 6171b8ff2bfe41d598b0cb2535799a11 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.746145 1173 ts_tablet_manager.cc:1428] T 6171b8ff2bfe41d598b0cb2535799a11 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:41.746363 1173 tablet_bootstrap.cc:492] T 79eb61cedd544c49869471eebe2fd237 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:41.747301 1181 tablet_bootstrap.cc:654] T 54d3664071b146a3ac7c6885f9bda789 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.748524 1177 tablet_bootstrap.cc:492] T fea2ef4e9a63491db850bb5393938017 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:41.748684 1177 ts_tablet_manager.cc:1397] T fea2ef4e9a63491db850bb5393938017 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.002s
I20250112 02:52:41.749342 1177 raft_consensus.cc:357] T fea2ef4e9a63491db850bb5393938017 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.749553 1177 raft_consensus.cc:383] T fea2ef4e9a63491db850bb5393938017 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.749627 1177 raft_consensus.cc:738] T fea2ef4e9a63491db850bb5393938017 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.749928 1177 consensus_queue.cc:260] T fea2ef4e9a63491db850bb5393938017 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.750468 1177 ts_tablet_manager.cc:1428] T fea2ef4e9a63491db850bb5393938017 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:41.751024 1173 tablet_bootstrap.cc:654] T 79eb61cedd544c49869471eebe2fd237 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.751159 2609 consensus_queue.cc:1035] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:41.757934 2451 raft_consensus.cc:491] T fea2ef4e9a63491db850bb5393938017 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.758036 2451 raft_consensus.cc:513] T fea2ef4e9a63491db850bb5393938017 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.758590 2451 leader_election.cc:290] T fea2ef4e9a63491db850bb5393938017 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:41.759398 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea2ef4e9a63491db850bb5393938017" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:41.759552 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea2ef4e9a63491db850bb5393938017" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:41.759615 1051 raft_consensus.cc:2388] T fea2ef4e9a63491db850bb5393938017 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 289afc116aa74408aa5bff74e11c50d8 in current term 1: Already voted for candidate fe36951a83fb4e57b26e187edbcbfe3d in this term.
I20250112 02:52:41.760143 714 leader_election.cc:304] T fea2ef4e9a63491db850bb5393938017 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:41.760440 2451 raft_consensus.cc:3054] T fea2ef4e9a63491db850bb5393938017 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.762872 754 tablet_service.cc:1514] Processing DeleteTablet for tablet b1d9fae290ec4281839991f9d40b105c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.763273 2451 raft_consensus.cc:2743] T fea2ef4e9a63491db850bb5393938017 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:41.764389 754 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e7882e07784bdda18dc1ffd20963c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.767779 754 tablet_service.cc:1514] Processing DeleteTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.777678 2562 ts_tablet_manager.cc:1918] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:41.777784 2562 log.cc:1198] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/3e378a6a9b4a455c9f5e1d1ec0fd6ca0
I20250112 02:52:41.778126 2562 ts_tablet_manager.cc:1939] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:41.783632 1181 tablet_bootstrap.cc:492] T 54d3664071b146a3ac7c6885f9bda789 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:41.783844 1181 ts_tablet_manager.cc:1397] T 54d3664071b146a3ac7c6885f9bda789 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20250112 02:52:41.784543 1181 raft_consensus.cc:357] T 54d3664071b146a3ac7c6885f9bda789 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.785596 1177 tablet_bootstrap.cc:492] T 2052b4536f9745f0a785f330d706c1cf P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:41.785889 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:41.786012 2745 consensus_queue.cc:1035] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:41.786993 1177 tablet_bootstrap.cc:654] T 2052b4536f9745f0a785f330d706c1cf P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.787770 1181 raft_consensus.cc:383] T 54d3664071b146a3ac7c6885f9bda789 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.787950 1181 raft_consensus.cc:738] T 54d3664071b146a3ac7c6885f9bda789 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.788328 1181 consensus_queue.cc:260] T 54d3664071b146a3ac7c6885f9bda789 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:41.788730 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet c29a9ca43e034b43a00207e5847bbee3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c29a9ca43e034b43a00207e5847bbee3 already in progress: creating tablet
W20250112 02:52:41.788967 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 because the tablet was not found. No further retry: Not found: Tablet not found: 3e378a6a9b4a455c9f5e1d1ec0fd6ca0
W20250112 02:52:41.789183 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet bab4978edea0445fb0456e88e7fb4c44 because the tablet was not found. No further retry: Not found: Tablet not found: bab4978edea0445fb0456e88e7fb4c44
W20250112 02:52:41.789355 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b1d9fae290ec4281839991f9d40b105c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1d9fae290ec4281839991f9d40b105c already in progress: creating tablet
I20250112 02:52:41.789352 1181 ts_tablet_manager.cc:1428] T 54d3664071b146a3ac7c6885f9bda789 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20250112 02:52:41.789515 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet a8e7882e07784bdda18dc1ffd20963c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8e7882e07784bdda18dc1ffd20963c8 already in progress: creating tablet
W20250112 02:52:41.789656 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f776e2a6ec704cf9b64483bc7d5b1dff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f776e2a6ec704cf9b64483bc7d5b1dff already in progress: creating tablet
I20250112 02:52:41.789768 1181 tablet_bootstrap.cc:492] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:41.791147 1173 tablet_bootstrap.cc:492] T 79eb61cedd544c49869471eebe2fd237 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:41.791400 1173 ts_tablet_manager.cc:1397] T 79eb61cedd544c49869471eebe2fd237 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.001s
I20250112 02:52:41.778825 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 255) took 1436 ms. Trace:
I20250112 02:52:41.791812 2562 rpcz_store.cc:276] 0112 02:52:40.342004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.342088 (+ 84us) service_pool.cc:225] Handling call
0112 02:52:41.721820 (+1379732us) ts_tablet_manager.cc:1145] Deleting tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0
0112 02:52:41.721827 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.734402 (+ 12575us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.777579 (+ 43177us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.778807 (+ 1228us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1379547}
I20250112 02:52:41.793078 1173 raft_consensus.cc:357] T 79eb61cedd544c49869471eebe2fd237 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.793542 1173 raft_consensus.cc:383] T 79eb61cedd544c49869471eebe2fd237 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.793754 1173 raft_consensus.cc:738] T 79eb61cedd544c49869471eebe2fd237 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.794250 1173 consensus_queue.cc:260] T 79eb61cedd544c49869471eebe2fd237 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.803833 2562 tablet_replica.cc:331] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:41.804003 2562 raft_consensus.cc:2238] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:41.803989 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]) successfully deleted
I20250112 02:52:41.804118 2562 raft_consensus.cc:2267] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:41.806479 2562 ts_tablet_manager.cc:1905] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:41.792800 1181 tablet_bootstrap.cc:654] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.818725 1177 tablet_bootstrap.cc:492] T 2052b4536f9745f0a785f330d706c1cf P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:41.818907 1177 ts_tablet_manager.cc:1397] T 2052b4536f9745f0a785f330d706c1cf P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.005s
I20250112 02:52:41.819612 1177 raft_consensus.cc:357] T 2052b4536f9745f0a785f330d706c1cf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.819849 1177 raft_consensus.cc:383] T 2052b4536f9745f0a785f330d706c1cf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.819929 1177 raft_consensus.cc:738] T 2052b4536f9745f0a785f330d706c1cf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.820161 1177 consensus_queue.cc:260] T 2052b4536f9745f0a785f330d706c1cf P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.820637 1177 ts_tablet_manager.cc:1428] T 2052b4536f9745f0a785f330d706c1cf P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:41.822507 1177 tablet_bootstrap.cc:492] T 79eb61cedd544c49869471eebe2fd237 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:41.823961 1177 tablet_bootstrap.cc:654] T 79eb61cedd544c49869471eebe2fd237 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.831269 1181 tablet_bootstrap.cc:492] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:41.831401 1181 ts_tablet_manager.cc:1397] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20250112 02:52:41.831987 1181 raft_consensus.cc:357] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.832188 1181 raft_consensus.cc:383] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.832247 1181 raft_consensus.cc:738] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.835578 1181 consensus_queue.cc:260] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.837775 2562 ts_tablet_manager.cc:1918] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:41.839531 2562 log.cc:1198] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/3b9280db29e54d41af1bed2fa5aa5535
I20250112 02:52:41.838910 1173 ts_tablet_manager.cc:1428] T 79eb61cedd544c49869471eebe2fd237 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.047s user 0.000s sys 0.003s
I20250112 02:52:41.839891 2562 ts_tablet_manager.cc:1939] T 3b9280db29e54d41af1bed2fa5aa5535 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:41.841328 2451 raft_consensus.cc:491] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.841435 2451 raft_consensus.cc:513] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.841976 2451 leader_election.cc:290] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:41.786743 452 catalog_manager.cc:5526] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:41.841313 2673 raft_consensus.cc:491] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.845240 2673 raft_consensus.cc:513] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.845737 2673 leader_election.cc:290] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:41.842563 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73f2fa452b84be398476b8aba6c9a08" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:41.845933 917 raft_consensus.cc:2463] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:41.846344 1181 ts_tablet_manager.cc:1428] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20250112 02:52:41.846673 713 leader_election.cc:304] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:41.847036 2451 raft_consensus.cc:2798] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:41.847097 2451 raft_consensus.cc:491] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:41.847141 2451 raft_consensus.cc:3054] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.847565 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c04f1a2cb04950a9f6170e252c19b4" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:41.849520 2451 raft_consensus.cc:513] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.851018 2451 leader_election.cc:290] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:41.848744 855 leader_election.cc:343] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.850790 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73f2fa452b84be398476b8aba6c9a08" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:41.851596 917 raft_consensus.cc:3054] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.852314 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 3b9280db29e54d41af1bed2fa5aa5535 (table default.Table_6671f628f89b428a83eca77e9ca26516 [id=e0577e1a40064437bb7eb4c66fbd914e]) successfully deleted
I20250112 02:52:41.852600 2562 tablet_replica.cc:331] T ff1b3ee229c445b19405cf4a19cf8003 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:41.852770 2562 raft_consensus.cc:2238] T ff1b3ee229c445b19405cf4a19cf8003 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:41.852880 2562 raft_consensus.cc:2267] T ff1b3ee229c445b19405cf4a19cf8003 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:41.853345 2562 ts_tablet_manager.cc:1905] T ff1b3ee229c445b19405cf4a19cf8003 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:41.853930 917 raft_consensus.cc:2463] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:41.854323 713 leader_election.cc:304] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:41.854406 1181 tablet_bootstrap.cc:492] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:41.854709 2451 raft_consensus.cc:2798] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:41.855379 2451 raft_consensus.cc:695] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:41.855724 1181 tablet_bootstrap.cc:654] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.855675 2451 consensus_queue.cc:237] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.861286 1177 tablet_bootstrap.cc:492] T 79eb61cedd544c49869471eebe2fd237 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:41.862994 1177 ts_tablet_manager.cc:1397] T 79eb61cedd544c49869471eebe2fd237 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20250112 02:52:41.863835 1177 raft_consensus.cc:357] T 79eb61cedd544c49869471eebe2fd237 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.864034 1177 raft_consensus.cc:383] T 79eb61cedd544c49869471eebe2fd237 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.864099 1177 raft_consensus.cc:738] T 79eb61cedd544c49869471eebe2fd237 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.864297 1177 consensus_queue.cc:260] T 79eb61cedd544c49869471eebe2fd237 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.864985 1177 ts_tablet_manager.cc:1428] T 79eb61cedd544c49869471eebe2fd237 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:41.865305 1177 tablet_bootstrap.cc:492] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:41.866727 1177 tablet_bootstrap.cc:654] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.882061 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c04f1a2cb04950a9f6170e252c19b4" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:41.882777 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73f2fa452b84be398476b8aba6c9a08" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:41.882948 1049 raft_consensus.cc:3054] T c73f2fa452b84be398476b8aba6c9a08 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.883715 1173 tablet_bootstrap.cc:492] T 37862023095644bab2403576e8cb8c58 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:41.885342 1049 raft_consensus.cc:2463] T c73f2fa452b84be398476b8aba6c9a08 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:41.891558 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73f2fa452b84be398476b8aba6c9a08" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:41.891709 1051 raft_consensus.cc:2371] T c73f2fa452b84be398476b8aba6c9a08 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1. Re-sending same reply.
I20250112 02:52:41.892406 2562 ts_tablet_manager.cc:1918] T ff1b3ee229c445b19405cf4a19cf8003 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:41.892609 2562 log.cc:1198] T ff1b3ee229c445b19405cf4a19cf8003 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/ff1b3ee229c445b19405cf4a19cf8003
W20250112 02:52:41.892623 854 leader_election.cc:343] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.892825 854 leader_election.cc:304] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:41.893255 2673 raft_consensus.cc:2743] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:41.893802 2562 ts_tablet_manager.cc:1939] T ff1b3ee229c445b19405cf4a19cf8003 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:41.894991 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet ff1b3ee229c445b19405cf4a19cf8003 (table default.Table_6b8cb70a5e5b4519af5d247f6b5bb8b0 [id=a28f5cfbcfe64768b40477ffc2053e27]) successfully deleted
W20250112 02:52:41.895771 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet d0cb54003a83472cb9c57ace3e65b0be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0cb54003a83472cb9c57ace3e65b0be already in progress: creating tablet
W20250112 02:52:41.896309 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet f776e2a6ec704cf9b64483bc7d5b1dff because the tablet was not found. No further retry: Not found: Tablet not found: f776e2a6ec704cf9b64483bc7d5b1dff
W20250112 02:52:41.896792 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 7a34d03a80ae402db6607261a5c40b60 because the tablet was not found. No further retry: Not found: Tablet not found: 7a34d03a80ae402db6607261a5c40b60
W20250112 02:52:41.897289 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet f31b777e7e0f43a28839d8c48c6a6590 because the tablet was not found. No further retry: Not found: Tablet not found: f31b777e7e0f43a28839d8c48c6a6590
I20250112 02:52:41.897043 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
W20250112 02:52:41.897755 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 9af55dab202444358ea1a0674d37946d because the tablet was not found. No further retry: Not found: Tablet not found: 9af55dab202444358ea1a0674d37946d
I20250112 02:52:41.898029 2562 tablet_replica.cc:331] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:41.898247 2562 raft_consensus.cc:2238] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:41.898408 2562 raft_consensus.cc:2267] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:41.898885 2562 ts_tablet_manager.cc:1905] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:41.904808 2673 raft_consensus.cc:491] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.904939 2673 raft_consensus.cc:513] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.905601 2673 leader_election.cc:290] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:41.906294 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500f0c5c76e54cd8808e6c8611e99d83" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:41.906818 855 leader_election.cc:343] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.907688 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500f0c5c76e54cd8808e6c8611e99d83" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:41.908190 854 leader_election.cc:343] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.908279 854 leader_election.cc:304] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:41.908618 2673 raft_consensus.cc:2743] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:41.925578 447 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 185) took 1252 ms. Trace:
I20250112 02:52:41.925731 447 rpcz_store.cc:276] 0112 02:52:40.673597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.619713 (+946116us) service_pool.cc:225] Handling call
0112 02:52:41.671978 (+ 52265us) spinlock_profiling.cc:243] Waited 51.3 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd72d03 00007f3e13e75236 00007f3e13e74f65 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:41.672710 (+ 732us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:41.673212 (+ 502us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:41.738661 (+ 65449us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:41.925427 (+186766us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:41.925536 (+ 109us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:41.673813 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.673900 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.673917 (+ 17us) write_op.cc:435] Acquired schema lock
0112 02:52:41.673919 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:41.673997 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.674006 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.674011 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:41.674077 (+ 66us) tablet.cc:698] Row locks acquired
0112 02:52:41.674080 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:41.674108 (+ 28us) write_op.cc:270] Start()
0112 02:52:41.674135 (+ 27us) write_op.cc:276] Timestamp: P: 1736650361673993 usec, L: 0
0112 02:52:41.674139 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:41.674279 (+ 140us) log.cc:844] Serialized 433 byte log entry
0112 02:52:41.735524 (+ 61245us) op_driver.cc:464] REPLICATION: finished
0112 02:52:41.737785 (+ 2261us) write_op.cc:301] APPLY: starting
0112 02:52:41.737795 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:41.737824 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:41.737847 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:41.737949 (+ 102us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:41.737974 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:41.737980 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:41.738043 (+ 63us) log.cc:844] Serialized 29 byte log entry
0112 02:52:41.738105 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.738117 (+ 12us) write_op.cc:454] Released schema lock
0112 02:52:41.738122 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:41.738204 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.738208 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":118089984,"child_traces":[["op",{"apply.queue_time_us":2211,"apply.run_cpu_time_us":429,"apply.run_wall_time_us":429,"num_ops":2,"prepare.queue_time_us":136,"prepare.run_cpu_time_us":534,"prepare.run_wall_time_us":535,"replication_time_us":61369}]]}
I20250112 02:52:41.926537 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_ef2001a2d9b24b4bb1f8fb9102703022" } new_table_name: "default.Table_e8cef38ebaa243ed83bfb41bcea1933f" modify_external_catalogs: true
I20250112 02:52:41.927520 2562 ts_tablet_manager.cc:1918] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:41.928189 2562 log.cc:1198] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/058a9ca089cd4967887b0a1127c7d03f
I20250112 02:52:41.928579 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet b124c15736084e6d8f69b4fb7c41cf39
I20250112 02:52:41.929289 454 catalog_manager.cc:6596] ReplaceTablet: tablet b124c15736084e6d8f69b4fb7c41cf39 deleted and replaced by tablet 97c7ce72d02049089c653684c53cb53c
I20250112 02:52:41.929389 454 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:40610 (request call id 187) took 1024 ms. Trace:
I20250112 02:52:41.929678 895 tablet_service.cc:1514] Processing DeleteTablet for tablet b124c15736084e6d8f69b4fb7c41cf39 with delete_type TABLET_DATA_DELETED (replaced by tablet 97c7ce72d02049089c653684c53cb53c) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:41.929469 454 rpcz_store.cc:276] 0112 02:52:40.904839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.640976 (+736137us) service_pool.cc:225] Handling call
0112 02:52:41.671728 (+ 30752us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd7f0ab 00007f3e13e81b77 00007f3e13e818a5 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:41.929360 (+257632us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:41.765069 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.765163 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.765167 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:41.765169 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:41.765240 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.765247 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.765251 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:41.765299 (+ 48us) tablet.cc:698] Row locks acquired
0112 02:52:41.765301 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:41.765321 (+ 20us) write_op.cc:270] Start()
0112 02:52:41.765340 (+ 19us) write_op.cc:276] Timestamp: P: 1736650361765280 usec, L: 0
0112 02:52:41.765343 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:41.765447 (+ 104us) log.cc:844] Serialized 622 byte log entry
0112 02:52:41.795921 (+ 30474us) op_driver.cc:464] REPLICATION: finished
0112 02:52:41.796581 (+ 660us) write_op.cc:301] APPLY: starting
0112 02:52:41.796590 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:41.796621 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:41.796642 (+ 21us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:41.796725 (+ 83us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:41.796770 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:41.796776 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:41.796822 (+ 46us) log.cc:844] Serialized 29 byte log entry
0112 02:52:41.796870 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.796882 (+ 12us) write_op.cc:454] Released schema lock
0112 02:52:41.796887 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:41.796937 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.796941 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":31808384,"child_traces":[["op",{"apply.queue_time_us":630,"apply.run_cpu_time_us":351,"apply.run_wall_time_us":351,"mutex_wait_us":6557,"num_ops":2,"prepare.queue_time_us":6612,"prepare.run_cpu_time_us":424,"prepare.run_wall_time_us":423,"replication_time_us":30560}]]}
I20250112 02:52:41.930159 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_56ea07abda424090a7942efabc871092" } modify_external_catalogs: true
I20250112 02:52:41.930348 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_56ea07abda424090a7942efabc871092" } modify_external_catalogs: true
I20250112 02:52:41.930500 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet b124c15736084e6d8f69b4fb7c41cf39 with delete_type TABLET_DATA_DELETED (replaced by tablet 97c7ce72d02049089c653684c53cb53c) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.931715 754 tablet_service.cc:1514] Processing DeleteTablet for tablet b124c15736084e6d8f69b4fb7c41cf39 with delete_type TABLET_DATA_DELETED (replaced by tablet 97c7ce72d02049089c653684c53cb53c) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.932348 450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 184) took 1350 ms. Trace:
I20250112 02:52:41.932483 450 rpcz_store.cc:276] 0112 02:52:40.582153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.599361 (+1017208us) service_pool.cc:225] Handling call
0112 02:52:41.674470 (+ 75109us) spinlock_profiling.cc:243] Waited 51.5 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd72d03 00007f3e13e75236 00007f3e13e74f65 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:41.675442 (+ 972us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:41.675856 (+ 414us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:41.745212 (+ 69356us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:41.932163 (+186951us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:41.932290 (+ 127us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:41.702443 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.702520 (+ 77us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.702525 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:41.702527 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:41.702592 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.702600 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.702604 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:41.702659 (+ 55us) tablet.cc:698] Row locks acquired
0112 02:52:41.702662 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:41.702685 (+ 23us) write_op.cc:270] Start()
0112 02:52:41.702708 (+ 23us) write_op.cc:276] Timestamp: P: 1736650361702654 usec, L: 0
0112 02:52:41.702711 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:41.702837 (+ 126us) log.cc:844] Serialized 433 byte log entry
0112 02:52:41.744559 (+ 41722us) op_driver.cc:464] REPLICATION: finished
0112 02:52:41.744724 (+ 165us) write_op.cc:301] APPLY: starting
0112 02:52:41.744735 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:41.744764 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:41.744794 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:41.744879 (+ 85us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:41.744903 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:41.744909 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:41.744957 (+ 48us) log.cc:844] Serialized 29 byte log entry
0112 02:52:41.745007 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.745022 (+ 15us) write_op.cc:454] Released schema lock
0112 02:52:41.745027 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:41.745092 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.745096 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":118410112,"child_traces":[["op",{"apply.queue_time_us":83,"apply.run_cpu_time_us":374,"apply.run_wall_time_us":374,"mutex_wait_us":2025,"num_ops":2,"prepare.queue_time_us":4091,"prepare.run_cpu_time_us":458,"prepare.run_wall_time_us":458,"replication_time_us":37782}]]}
I20250112 02:52:41.933416 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_39f67a35c5c640dd8cbbcea89df5e4fc" } new_table_name: "default.Table_5aeb2d7bb68c46a589a48526dcc40aaf" modify_external_catalogs: true
I20250112 02:52:41.934911 446 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 183) took 1355 ms. Trace:
I20250112 02:52:41.935016 446 rpcz_store.cc:276] 0112 02:52:40.579848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.511905 (+932057us) service_pool.cc:225] Handling call
0112 02:52:41.620121 (+108216us) spinlock_profiling.cc:243] Waited 24.1 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd72d03 00007f3e13e75236 00007f3e13e74f65 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:41.684850 (+ 64729us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:41.685245 (+ 395us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:41.740036 (+ 54791us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:41.934798 (+194762us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:41.934859 (+ 61us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:41.701833 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.701928 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.701934 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:41.701937 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:41.702009 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.702017 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.702021 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:41.702156 (+ 135us) tablet.cc:698] Row locks acquired
0112 02:52:41.702159 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:41.702186 (+ 27us) write_op.cc:270] Start()
0112 02:52:41.702213 (+ 27us) write_op.cc:276] Timestamp: P: 1736650361702154 usec, L: 0
0112 02:52:41.702217 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:41.702348 (+ 131us) log.cc:844] Serialized 433 byte log entry
0112 02:52:41.739318 (+ 36970us) op_driver.cc:464] REPLICATION: finished
0112 02:52:41.739642 (+ 324us) write_op.cc:301] APPLY: starting
0112 02:52:41.739653 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:41.739701 (+ 48us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:41.739719 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:41.739786 (+ 67us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:41.739807 (+ 21us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:41.739812 (+ 5us) write_op.cc:312] APPLY: finished
0112 02:52:41.739854 (+ 42us) log.cc:844] Serialized 29 byte log entry
0112 02:52:41.739897 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.739908 (+ 11us) write_op.cc:454] Released schema lock
0112 02:52:41.739913 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":55458048,"child_traces":[["op",{"apply.queue_time_us":176,"apply.run_cpu_time_us":317,"apply.run_wall_time_us":317,"mutex_wait_us":14508,"num_ops":2,"prepare.queue_time_us":15997,"prepare.run_cpu_time_us":559,"prepare.run_wall_time_us":576,"replication_time_us":37026}]]}
I20250112 02:52:41.935774 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_b2879d6a85294514989d7b68d3ac97d4" } new_table_name: "default.Table_d22db1cc1d354af495d9888de1bfc22a" modify_external_catalogs: true
I20250112 02:52:41.937284 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 795ac7eaf81d48bb926865d9ebb388b6
I20250112 02:52:41.938006 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 795ac7eaf81d48bb926865d9ebb388b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 818cd6362b94473ea06ed68fa5ffd1bc) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:41.943068 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 795ac7eaf81d48bb926865d9ebb388b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 818cd6362b94473ea06ed68fa5ffd1bc) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:41.943280 455 catalog_manager.cc:6596] ReplaceTablet: tablet 795ac7eaf81d48bb926865d9ebb388b6 deleted and replaced by tablet 818cd6362b94473ea06ed68fa5ffd1bc
I20250112 02:52:41.943442 455 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:40610 (request call id 186) took 1184 ms. Trace:
I20250112 02:52:41.943424 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 795ac7eaf81d48bb926865d9ebb388b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 818cd6362b94473ea06ed68fa5ffd1bc) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:41.943674 2003 tablet_replica.cc:331] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:41.943553 455 rpcz_store.cc:276] 0112 02:52:40.759238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.637777 (+878539us) service_pool.cc:225] Handling call
0112 02:52:41.669962 (+ 32185us) spinlock_profiling.cc:243] Waited 23.2 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd7f0ab 00007f3e13e81b77 00007f3e13e818a5 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:41.943399 (+273437us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:41.670848 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.670953 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.670958 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:41.670960 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:41.671080 (+ 120us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.671093 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.671097 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:41.671172 (+ 75us) tablet.cc:698] Row locks acquired
0112 02:52:41.671175 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:41.671211 (+ 36us) write_op.cc:270] Start()
0112 02:52:41.671241 (+ 30us) write_op.cc:276] Timestamp: P: 1736650361671093 usec, L: 0
0112 02:52:41.671245 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:41.671420 (+ 175us) log.cc:844] Serialized 656 byte log entry
0112 02:52:41.735421 (+ 64001us) op_driver.cc:464] REPLICATION: finished
0112 02:52:41.737228 (+ 1807us) write_op.cc:301] APPLY: starting
0112 02:52:41.737245 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:41.737287 (+ 42us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:41.737315 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:41.737410 (+ 95us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:41.737473 (+ 63us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:41.737481 (+ 8us) write_op.cc:312] APPLY: finished
0112 02:52:41.737543 (+ 62us) log.cc:844] Serialized 29 byte log entry
0112 02:52:41.737615 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.737631 (+ 16us) write_op.cc:454] Released schema lock
0112 02:52:41.737638 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:41.737704 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.737708 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":53348736,"child_traces":[["op",{"apply.queue_time_us":1749,"apply.run_cpu_time_us":472,"apply.run_wall_time_us":472,"num_ops":2,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":671,"prepare.run_wall_time_us":676,"replication_time_us":64135}]]}
I20250112 02:52:41.943920 2003 raft_consensus.cc:2238] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:41.944077 2003 raft_consensus.cc:2267] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:41.944133 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_cd1f185d82b3474083246ec3702851fc" } modify_external_catalogs: true
I20250112 02:52:41.944268 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_cd1f185d82b3474083246ec3702851fc" } modify_external_catalogs: true
I20250112 02:52:41.944669 2003 ts_tablet_manager.cc:1905] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:41.946523 2451 raft_consensus.cc:491] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.946631 2451 raft_consensus.cc:513] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.947189 2451 leader_election.cc:290] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:41.947671 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e477fc157aed4563923d85d66014293a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:41.947873 918 raft_consensus.cc:2388] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 289afc116aa74408aa5bff74e11c50d8 in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
I20250112 02:52:41.948576 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e477fc157aed4563923d85d66014293a" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:41.949227 714 leader_election.cc:304] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:41.949820 2179 raft_consensus.cc:3054] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:41.952986 2179 raft_consensus.cc:2743] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:41.953599 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet b124c15736084e6d8f69b4fb7c41cf39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b124c15736084e6d8f69b4fb7c41cf39 already in progress: creating tablet
I20250112 02:52:41.956800 2208 consensus_queue.cc:1035] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:41.957731 2003 ts_tablet_manager.cc:1918] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:41.957826 2003 log.cc:1198] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/795ac7eaf81d48bb926865d9ebb388b6
I20250112 02:52:41.957916 1017 tablet_service.cc:1467] Processing CreateTablet for tablet d1269a6ed80e46ecb7f4f87e5afb732f (DEFAULT_TABLE table=default.Table_9fdbd960fe3a41c59ab9262ef27fcf95 [id=16ed56130f1e4e03baad8171ff45745c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:41.958194 2003 ts_tablet_manager.cc:1939] T 795ac7eaf81d48bb926865d9ebb388b6 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:41.958271 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1269a6ed80e46ecb7f4f87e5afb732f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:41.963654 2562 ts_tablet_manager.cc:1939] T 058a9ca089cd4967887b0a1127c7d03f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:41.964136 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 795ac7eaf81d48bb926865d9ebb388b6 (table default.Table_178713b464bb4f16970851646a710574 [id=d3f31247f0e848f8b2dfa18f3437ca63]) successfully deleted
I20250112 02:52:41.964334 754 tablet_service.cc:1467] Processing CreateTablet for tablet d1269a6ed80e46ecb7f4f87e5afb732f (DEFAULT_TABLE table=default.Table_9fdbd960fe3a41c59ab9262ef27fcf95 [id=16ed56130f1e4e03baad8171ff45745c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:41.964637 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1269a6ed80e46ecb7f4f87e5afb732f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:41.967010 895 tablet_service.cc:1467] Processing CreateTablet for tablet d1269a6ed80e46ecb7f4f87e5afb732f (DEFAULT_TABLE table=default.Table_9fdbd960fe3a41c59ab9262ef27fcf95 [id=16ed56130f1e4e03baad8171ff45745c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:41.967360 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1269a6ed80e46ecb7f4f87e5afb732f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:41.968648 1173 tablet_bootstrap.cc:654] T 37862023095644bab2403576e8cb8c58 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:41.969347 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 058a9ca089cd4967887b0a1127c7d03f (table default.Table_a56064e2c6b449b7b1e32f2a69f2f7fe [id=448877a19b0d4ed2abbc3a796e40eaeb]) successfully deleted
W20250112 02:52:41.970546 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet f3771166b77e4f90aaa116ab1a9dc299 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3771166b77e4f90aaa116ab1a9dc299 already in progress: creating tablet
W20250112 02:52:41.971541 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 681a930541a84f0197ecafe670278e70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 681a930541a84f0197ecafe670278e70 already in progress: creating tablet
I20250112 02:52:41.972163 2562 tablet_replica.cc:331] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:41.972862 2562 raft_consensus.cc:2238] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:41.974632 2562 raft_consensus.cc:2267] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:41.975562 2562 ts_tablet_manager.cc:1905] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:41.900075 452 catalog_manager.cc:5526] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:41.977272 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:41.977735 1181 tablet_bootstrap.cc:492] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:41.977577 452 catalog_manager.cc:5526] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:41.977964 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 0882ea69a90b4ce3abb4f2da6475972f (DEFAULT_TABLE table=default.Table_4770176caf32467daca85ee46a034acd [id=45d7bbac986d4db9a3e0b6f1a88ffa1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:41.978101 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:41.978498 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0882ea69a90b4ce3abb4f2da6475972f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:41.978601 452 catalog_manager.cc:5526] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:41.979259 452 catalog_manager.cc:5526] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:41.979714 452 catalog_manager.cc:5526] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:41.980137 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:41.980448 452 catalog_manager.cc:5526] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:41.980644 2673 raft_consensus.cc:491] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:41.980762 2673 raft_consensus.cc:513] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.981355 2673 leader_election.cc:290] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:41.982020 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a32b424ec4431d88f6af45dd09089f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:41.982560 854 leader_election.cc:343] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.982789 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a32b424ec4431d88f6af45dd09089f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:41.983321 855 leader_election.cc:343] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:41.983417 855 leader_election.cc:304] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:41.984002 2673 raft_consensus.cc:2743] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:41.984037 754 tablet_service.cc:1467] Processing CreateTablet for tablet 0882ea69a90b4ce3abb4f2da6475972f (DEFAULT_TABLE table=default.Table_4770176caf32467daca85ee46a034acd [id=45d7bbac986d4db9a3e0b6f1a88ffa1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:41.984339 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0882ea69a90b4ce3abb4f2da6475972f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:41.984807 895 tablet_service.cc:1467] Processing CreateTablet for tablet 0882ea69a90b4ce3abb4f2da6475972f (DEFAULT_TABLE table=default.Table_4770176caf32467daca85ee46a034acd [id=45d7bbac986d4db9a3e0b6f1a88ffa1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:41.985316 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0882ea69a90b4ce3abb4f2da6475972f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:41.988849 2024 ts_tablet_manager.cc:1918] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:41.988982 2024 log.cc:1198] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/c4a3486ef011492c8a1df8d6278adedf
I20250112 02:52:41.989362 2024 ts_tablet_manager.cc:1939] T c4a3486ef011492c8a1df8d6278adedf P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:41.989836 1177 tablet_bootstrap.cc:492] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:41.989996 1177 ts_tablet_manager.cc:1397] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.125s user 0.010s sys 0.012s
I20250112 02:52:41.990190 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 237) took 2985 ms. Trace:
I20250112 02:52:41.990268 2024 rpcz_store.cc:276] 0112 02:52:39.004367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.005357 (+ 990us) service_pool.cc:225] Handling call
0112 02:52:41.733904 (+2728547us) ts_tablet_manager.cc:1145] Deleting tablet c4a3486ef011492c8a1df8d6278adedf
0112 02:52:41.734354 (+ 450us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.778689 (+ 44335us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.988759 (+210070us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:41.990172 (+ 1413us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2728408}
I20250112 02:52:41.990499 897 tablet_service.cc:1467] Processing CreateTablet for tablet ab183c43f6a64270ad744e61364f0a99 (DEFAULT_TABLE table=default.Table_9afc6941825a45bc8f98bf373ff5f27c [id=c76eb62dc8bf4b97b675787374189057]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:41.990746 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 245) took 2962 ms. Trace:
I20250112 02:52:41.990828 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab183c43f6a64270ad744e61364f0a99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:41.990809 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet c4a3486ef011492c8a1df8d6278adedf (table default.Table_33b6530b130041f0a2a4a7240e92ff15 [id=1d614fd3947a43bcac5a3d40cad6d80b]) successfully deleted
I20250112 02:52:41.990821 2024 rpcz_store.cc:276] 0112 02:52:39.028789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.028837 (+ 48us) service_pool.cc:225] Handling call
0112 02:52:41.990655 (+2961818us) ts_tablet_manager.cc:1145] Deleting tablet 49bb6d20cfa0453e90e048825114525e
0112 02:52:41.990665 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.990731 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2838549}
I20250112 02:52:41.991158 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 249) took 2838 ms. Trace:
W20250112 02:52:41.991271 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 49bb6d20cfa0453e90e048825114525e because the tablet was not found. No further retry: Not found: Tablet not found: 49bb6d20cfa0453e90e048825114525e
I20250112 02:52:41.991228 2024 rpcz_store.cc:276] 0112 02:52:39.152874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.153653 (+ 779us) service_pool.cc:225] Handling call
0112 02:52:41.991069 (+2837416us) ts_tablet_manager.cc:1145] Deleting tablet 64707ee432fb4e96a3bbf7252dfa1813
0112 02:52:41.991077 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.991145 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2837232}
I20250112 02:52:41.991564 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 250) took 2838 ms. Trace:
W20250112 02:52:41.991695 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 64707ee432fb4e96a3bbf7252dfa1813 because the tablet was not found. No further retry: Not found: Tablet not found: 64707ee432fb4e96a3bbf7252dfa1813
I20250112 02:52:41.991631 2024 rpcz_store.cc:276] 0112 02:52:39.152968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.153882 (+ 914us) service_pool.cc:225] Handling call
0112 02:52:41.991432 (+2837550us) ts_tablet_manager.cc:1145] Deleting tablet 3fd498ccce484a89855c982762086890
0112 02:52:41.991440 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.991549 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2837425}
I20250112 02:52:41.991897 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 251) took 2838 ms. Trace:
W20250112 02:52:41.992015 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 3fd498ccce484a89855c982762086890 because the tablet was not found. No further retry: Not found: Tablet not found: 3fd498ccce484a89855c982762086890
I20250112 02:52:41.991973 2024 rpcz_store.cc:276] 0112 02:52:39.153048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.154043 (+ 995us) service_pool.cc:225] Handling call
0112 02:52:41.991828 (+2837785us) ts_tablet_manager.cc:1145] Deleting tablet c04b64bc2f8348a086ed3c4e59f09b16
0112 02:52:41.991835 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.991884 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2837645}
I20250112 02:52:41.992249 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 248) took 2839 ms. Trace:
W20250112 02:52:41.992367 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet c04b64bc2f8348a086ed3c4e59f09b16 because the tablet was not found. No further retry: Not found: Tablet not found: c04b64bc2f8348a086ed3c4e59f09b16
I20250112 02:52:41.992311 2024 rpcz_store.cc:276] 0112 02:52:39.152787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.154810 (+ 2023us) service_pool.cc:225] Handling call
0112 02:52:41.992176 (+2837366us) ts_tablet_manager.cc:1145] Deleting tablet 97b31ebe035c4552baaffc3133ec73c1
0112 02:52:41.992187 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.992235 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2837169}
I20250112 02:52:41.992585 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 257) took 2836 ms. Trace:
W20250112 02:52:41.992678 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 97b31ebe035c4552baaffc3133ec73c1 because the tablet was not found. No further retry: Not found: Tablet not found: 97b31ebe035c4552baaffc3133ec73c1
I20250112 02:52:41.992650 2024 rpcz_store.cc:276] 0112 02:52:39.155617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.155706 (+ 89us) service_pool.cc:225] Handling call
0112 02:52:41.992512 (+2836806us) ts_tablet_manager.cc:1145] Deleting tablet fb0aba0139cf479b80b32f3dfc936535
0112 02:52:41.992521 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:41.992572 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2836664}
I20250112 02:52:41.992885 2024 tablet_replica.cc:331] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:41.993077 2024 raft_consensus.cc:2238] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:41.993232 2024 raft_consensus.cc:2267] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:41.993274 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet fb0aba0139cf479b80b32f3dfc936535 because the tablet was not found. No further retry: Not found: Tablet not found: fb0aba0139cf479b80b32f3dfc936535
I20250112 02:52:41.993702 2024 ts_tablet_manager.cc:1905] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:41.993652 1177 raft_consensus.cc:357] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.993800 1177 raft_consensus.cc:383] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.993842 1177 raft_consensus.cc:738] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.994029 1177 consensus_queue.cc:260] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.994308 1181 ts_tablet_manager.cc:1397] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.140s user 0.016s sys 0.006s
I20250112 02:52:41.994892 1181 raft_consensus.cc:357] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:41.995033 1181 raft_consensus.cc:383] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.995072 1181 raft_consensus.cc:738] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.996997 1173 tablet_bootstrap.cc:492] T 37862023095644bab2403576e8cb8c58 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:41.997095 1173 ts_tablet_manager.cc:1397] T 37862023095644bab2403576e8cb8c58 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.114s user 0.003s sys 0.001s
I20250112 02:52:41.997951 1173 raft_consensus.cc:357] T 37862023095644bab2403576e8cb8c58 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.998111 1173 raft_consensus.cc:383] T 37862023095644bab2403576e8cb8c58 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:41.998198 1173 raft_consensus.cc:738] T 37862023095644bab2403576e8cb8c58 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:41.998396 1173 consensus_queue.cc:260] T 37862023095644bab2403576e8cb8c58 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:41.998855 1173 ts_tablet_manager.cc:1428] T 37862023095644bab2403576e8cb8c58 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:42.002931 1031 tablet_service.cc:1467] Processing CreateTablet for tablet ab183c43f6a64270ad744e61364f0a99 (DEFAULT_TABLE table=default.Table_9afc6941825a45bc8f98bf373ff5f27c [id=c76eb62dc8bf4b97b675787374189057]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.003024 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:42.005875 772 raft_consensus.cc:3054] T cf54f5185a0f4c70b9057309fca14e56 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.006188 1181 consensus_queue.cc:260] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.010064 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab183c43f6a64270ad744e61364f0a99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.006611 754 tablet_service.cc:1467] Processing CreateTablet for tablet ab183c43f6a64270ad744e61364f0a99 (DEFAULT_TABLE table=default.Table_9afc6941825a45bc8f98bf373ff5f27c [id=c76eb62dc8bf4b97b675787374189057]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.011701 2326 consensus_queue.cc:1035] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:42.012037 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab183c43f6a64270ad744e61364f0a99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.012539 1181 ts_tablet_manager.cc:1428] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250112 02:52:42.012550 452 catalog_manager.cc:5526] T a437952889494b0e820e41fdfffe708d P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:42.015018 1177 ts_tablet_manager.cc:1428] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20250112 02:52:42.025607 1173 tablet_bootstrap.cc:492] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:42.020174 2673 raft_consensus.cc:491] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.027182 2673 raft_consensus.cc:513] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.027889 2673 leader_election.cc:290] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:42.032270 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79eb61cedd544c49869471eebe2fd237" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:42.031862 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79eb61cedd544c49869471eebe2fd237" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:42.032444 1051 raft_consensus.cc:2463] T 79eb61cedd544c49869471eebe2fd237 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:42.032469 2562 ts_tablet_manager.cc:1918] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:42.032567 2562 log.cc:1198] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/8429a3d8636d4e2ca4c1191f85b9266b
I20250112 02:52:42.032568 771 raft_consensus.cc:2463] T 79eb61cedd544c49869471eebe2fd237 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:42.032963 855 leader_election.cc:304] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:42.033231 2673 raft_consensus.cc:2798] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:42.033296 2673 raft_consensus.cc:491] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:42.033365 2673 raft_consensus.cc:3054] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.033535 1177 tablet_bootstrap.cc:492] T 37862023095644bab2403576e8cb8c58 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:42.033910 1181 tablet_bootstrap.cc:492] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:42.035765 2673 raft_consensus.cc:513] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.036846 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79eb61cedd544c49869471eebe2fd237" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:42.037092 771 raft_consensus.cc:3054] T 79eb61cedd544c49869471eebe2fd237 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.038518 2673 leader_election.cc:290] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:42.039155 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79eb61cedd544c49869471eebe2fd237" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:42.039327 1051 raft_consensus.cc:3054] T 79eb61cedd544c49869471eebe2fd237 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.041728 1051 raft_consensus.cc:2463] T 79eb61cedd544c49869471eebe2fd237 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:42.042074 854 leader_election.cc:304] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:42.042356 2673 raft_consensus.cc:2798] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:42.042466 2673 raft_consensus.cc:695] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:42.042682 2673 consensus_queue.cc:237] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.032958 2562 ts_tablet_manager.cc:1939] T 8429a3d8636d4e2ca4c1191f85b9266b P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:42.044166 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 8429a3d8636d4e2ca4c1191f85b9266b (table default.Table_328664e3c01f483ca8e5ff50a497f68e [id=1f5c5f9f5e664250898a3935787b6b51]) successfully deleted
W20250112 02:52:42.044750 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet f924459d1150472aafb325a11825db23 because the tablet was not found. No further retry: Not found: Tablet not found: f924459d1150472aafb325a11825db23
I20250112 02:52:42.045145 2562 tablet_replica.cc:331] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:42.045356 2562 raft_consensus.cc:2238] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:42.045514 2562 raft_consensus.cc:2267] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:42.046535 2562 ts_tablet_manager.cc:1905] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:42.048389 1173 tablet_bootstrap.cc:654] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.052578 2024 ts_tablet_manager.cc:1918] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:42.054415 1173 tablet_bootstrap.cc:492] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:42.055181 1173 ts_tablet_manager.cc:1397] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.002s
I20250112 02:52:42.056303 1173 raft_consensus.cc:357] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.061841 1173 raft_consensus.cc:383] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.062119 1173 raft_consensus.cc:738] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.062518 1173 consensus_queue.cc:260] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.063637 1173 ts_tablet_manager.cc:1428] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250112 02:52:42.069257 2562 ts_tablet_manager.cc:1918] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:42.070341 2562 log.cc:1198] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/83c04f1a2cb04950a9f6170e252c19b4
I20250112 02:52:42.070855 2562 ts_tablet_manager.cc:1939] T 83c04f1a2cb04950a9f6170e252c19b4 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:42.070964 2024 log.cc:1198] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/b6d84027ad6d40a5be5afcdd854431a5
I20250112 02:52:42.071321 2024 ts_tablet_manager.cc:1939] T b6d84027ad6d40a5be5afcdd854431a5 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:42.072564 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 83c04f1a2cb04950a9f6170e252c19b4 (table default.Table_b1a770633f004b01b66ddfa83e0fd698 [id=20fddd79df734f3bb4552ac2ca4958e7]) successfully deleted
W20250112 02:52:42.074113 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet cbe30f9860e2492b90e8cedf8ff9daba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbe30f9860e2492b90e8cedf8ff9daba already in progress: creating tablet
W20250112 02:52:42.074914 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 76009bac22ce4136b148b493a004a0f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76009bac22ce4136b148b493a004a0f7 already in progress: creating tablet
W20250112 02:52:42.075954 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 940affdc2c484b7d9fd2de17f9d1cea7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 940affdc2c484b7d9fd2de17f9d1cea7 already in progress: creating tablet
W20250112 02:52:42.076534 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 13578542e9e741e8817e0ee63b10f1a1 because the tablet was not found. No further retry: Not found: Tablet not found: 13578542e9e741e8817e0ee63b10f1a1
I20250112 02:52:42.077039 2562 tablet_replica.cc:331] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:42.077507 2562 raft_consensus.cc:2238] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:42.078341 2562 raft_consensus.cc:2267] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:42.078964 2562 ts_tablet_manager.cc:1905] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:42.083807 1181 tablet_bootstrap.cc:654] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.086351 2562 ts_tablet_manager.cc:1918] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:42.086565 2562 log.cc:1198] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/b124c15736084e6d8f69b4fb7c41cf39
I20250112 02:52:42.086453 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:42.087004 2562 ts_tablet_manager.cc:1939] T b124c15736084e6d8f69b4fb7c41cf39 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:42.087255 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:42.088382 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet b124c15736084e6d8f69b4fb7c41cf39 (table default.Table_f0fb142b9aab490bb067d3cf64dc47b1 [id=1991c0821ef4478eb0eb9282ab772c0c]) successfully deleted
I20250112 02:52:42.088776 2562 tablet_replica.cc:331] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:42.089040 2562 raft_consensus.cc:2238] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:42.089272 2562 raft_consensus.cc:2267] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:42.089598 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:42.089932 2562 ts_tablet_manager.cc:1905] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:42.090149 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:42.090715 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_5d473cabff2e46e0b531fbce28d9d558 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 1 got 0
I20250112 02:52:42.093688 1177 tablet_bootstrap.cc:654] T 37862023095644bab2403576e8cb8c58 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.093973 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:42.096465 771 raft_consensus.cc:2463] T 79eb61cedd544c49869471eebe2fd237 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:42.097801 2562 ts_tablet_manager.cc:1918] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:42.097911 2562 log.cc:1198] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/795ac7eaf81d48bb926865d9ebb388b6
I20250112 02:52:42.098238 2562 ts_tablet_manager.cc:1939] T 795ac7eaf81d48bb926865d9ebb388b6 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:42.098790 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 795ac7eaf81d48bb926865d9ebb388b6 (table default.Table_178713b464bb4f16970851646a710574 [id=d3f31247f0e848f8b2dfa18f3437ca63]) successfully deleted
I20250112 02:52:42.101060 2787 raft_consensus.cc:491] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.101176 2787 raft_consensus.cc:513] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.101719 2787 leader_election.cc:290] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:42.102290 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f16ec377f01484eae10e63490da143f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:42.102840 854 leader_election.cc:343] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.103818 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f16ec377f01484eae10e63490da143f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:42.104362 855 leader_election.cc:343] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.104465 855 leader_election.cc:304] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:42.104841 2787 raft_consensus.cc:2743] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:42.105690 1181 tablet_bootstrap.cc:492] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:42.105845 1181 ts_tablet_manager.cc:1397] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.072s user 0.002s sys 0.000s
I20250112 02:52:42.106492 1181 raft_consensus.cc:357] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.106709 1181 raft_consensus.cc:383] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.106791 1181 raft_consensus.cc:738] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.107086 1181 consensus_queue.cc:260] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.107833 1181 ts_tablet_manager.cc:1428] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:42.108080 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 258) took 2952 ms. Trace:
I20250112 02:52:42.108156 1181 tablet_bootstrap.cc:492] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:42.108181 2024 rpcz_store.cc:276] 0112 02:52:39.155711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.155875 (+ 164us) service_pool.cc:225] Handling call
0112 02:52:41.992841 (+2836966us) ts_tablet_manager.cc:1145] Deleting tablet b6d84027ad6d40a5be5afcdd854431a5
0112 02:52:41.992849 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.049604 (+ 56755us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.052505 (+ 2901us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.108065 (+ 55560us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2836819}
I20250112 02:52:42.108554 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 260) took 2952 ms. Trace:
I20250112 02:52:42.108629 2024 rpcz_store.cc:276] 0112 02:52:39.155881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.156124 (+ 243us) service_pool.cc:225] Handling call
0112 02:52:42.108450 (+2952326us) ts_tablet_manager.cc:1145] Deleting tablet 04f135b226774285a077e5806131d91b
0112 02:52:42.108460 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.108540 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2952173}
I20250112 02:52:42.108682 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet b6d84027ad6d40a5be5afcdd854431a5 (table default.Table_31424b670f274fd3abd17eb8451ba95c [id=ebc9aae273fb4bcd8bd904f2179f1dd4]) successfully deleted
I20250112 02:52:42.108879 2024 tablet_replica.cc:331] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:42.109050 2024 raft_consensus.cc:2238] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250112 02:52:42.109151 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 04f135b226774285a077e5806131d91b because the tablet was not found. No further retry: Not found: Tablet not found: 04f135b226774285a077e5806131d91b
I20250112 02:52:42.109225 2024 raft_consensus.cc:2267] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:42.109876 2024 ts_tablet_manager.cc:1905] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:42.111555 1181 tablet_bootstrap.cc:654] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.115145 1173 tablet_bootstrap.cc:492] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:42.115566 2445 raft_consensus.cc:491] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.115697 2445 raft_consensus.cc:513] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.116324 2445 leader_election.cc:290] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.116672 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6171b8ff2bfe41d598b0cb2535799a11" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:42.116853 1051 raft_consensus.cc:2463] T 6171b8ff2bfe41d598b0cb2535799a11 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:42.117204 714 leader_election.cc:304] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:42.117439 2445 raft_consensus.cc:2798] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:42.117509 2445 raft_consensus.cc:491] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:42.117558 1173 tablet_bootstrap.cc:654] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.117576 2445 raft_consensus.cc:3054] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.119948 2445 raft_consensus.cc:513] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.120401 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:42.120539 2445 leader_election.cc:290] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.120931 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6171b8ff2bfe41d598b0cb2535799a11" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:42.121106 1051 raft_consensus.cc:3054] T 6171b8ff2bfe41d598b0cb2535799a11 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.123935 1051 raft_consensus.cc:2463] T 6171b8ff2bfe41d598b0cb2535799a11 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:42.124393 714 leader_election.cc:304] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:42.124604 2445 raft_consensus.cc:2798] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:42.124725 2445 raft_consensus.cc:695] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:42.124961 2445 consensus_queue.cc:237] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.117744 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6171b8ff2bfe41d598b0cb2535799a11" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:42.125612 918 raft_consensus.cc:2463] T 6171b8ff2bfe41d598b0cb2535799a11 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:42.123579 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6171b8ff2bfe41d598b0cb2535799a11" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:42.126199 917 raft_consensus.cc:3054] T 6171b8ff2bfe41d598b0cb2535799a11 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.128535 917 raft_consensus.cc:2463] T 6171b8ff2bfe41d598b0cb2535799a11 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:42.125463 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3dab0466476c4a9aa265271c1abd91a4 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:42.132320 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_a9907ed8bfe14625977ec98b67e9d599 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 2 got 0
I20250112 02:52:42.132499 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_b9dff6cb7bbd4d0a88de458399b0d2c0 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 2 got 0
I20250112 02:52:42.132758 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ca2165362ee541adac724b63f39b1489 (table default.Table_41b7056b6e9240928d0af52aec2e9338 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 2 got 0
I20250112 02:52:42.133221 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:42.136605 1177 tablet_bootstrap.cc:492] T 37862023095644bab2403576e8cb8c58 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:42.136801 1177 ts_tablet_manager.cc:1397] T 37862023095644bab2403576e8cb8c58 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.103s user 0.001s sys 0.002s
I20250112 02:52:42.139626 1177 raft_consensus.cc:357] T 37862023095644bab2403576e8cb8c58 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.139874 1177 raft_consensus.cc:383] T 37862023095644bab2403576e8cb8c58 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.139961 1177 raft_consensus.cc:738] T 37862023095644bab2403576e8cb8c58 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.140286 2024 ts_tablet_manager.cc:1918] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:42.140379 2024 log.cc:1198] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/1dda52f012794493a102ed1abc718f5b
I20250112 02:52:42.140233 1177 consensus_queue.cc:260] T 37862023095644bab2403576e8cb8c58 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.140700 2024 ts_tablet_manager.cc:1939] T 1dda52f012794493a102ed1abc718f5b P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:42.141088 1177 ts_tablet_manager.cc:1428] T 37862023095644bab2403576e8cb8c58 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:42.144168 449 catalog_manager.cc:5526] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:42.144873 1177 tablet_bootstrap.cc:492] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:42.145334 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:42.145539 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:42.146063 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:42.146479 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:42.146986 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet d95f98ae7bdf42518bbff55e257fea88 (table default.Table_2fcdbb05970f4a098f7ba3e287fa0348 [id=70c61753e9944e1fbd65e9ec41a48ee2]). Expected version 1 got 0
I20250112 02:52:42.147320 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:42.147567 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:42.148030 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:42.148532 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:42.148731 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:42.149401 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:42.149555 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:42.149696 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 17c54c5dc86446cf85974777c8145ef6 (table default.Table_8ab566cf9c8f45eba4dc9bb85264c823 [id=a622f7a973344a388b3275adbcce6234]). Expected version 1 got 0
I20250112 02:52:42.149816 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
W20250112 02:52:42.170097 989 consensus_peers.cc:487] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 795ac7eaf81d48bb926865d9ebb388b6. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:42.176462 2673 raft_consensus.cc:491] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.176648 2673 raft_consensus.cc:513] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.178078 2673 leader_election.cc:290] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:42.178690 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea18666ec324403b9f10a099688438b8" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:42.179353 854 leader_election.cc:343] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.185703 2789 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 436220c23fa948efb5b6ca15875d8b0e. Latest consensus state: current_term: 4 leader_uuid: "436220c23fa948efb5b6ca15875d8b0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:42.185971 2789 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:52:42.191768 2840 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "436220c23fa948efb5b6ca15875d8b0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:42.192937 2840 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:52:42.195995 1173 tablet_bootstrap.cc:492] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:42.196125 1173 ts_tablet_manager.cc:1397] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.081s user 0.003s sys 0.000s
I20250112 02:52:42.196769 1173 raft_consensus.cc:357] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.196964 1173 raft_consensus.cc:383] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.197019 1173 raft_consensus.cc:738] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.197192 1173 consensus_queue.cc:260] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.199363 1173 ts_tablet_manager.cc:1428] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:42.200625 2283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44576 (request call id 0) took 5011 ms. Trace:
I20250112 02:52:42.208489 1173 tablet_bootstrap.cc:492] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:42.209837 1173 tablet_bootstrap.cc:654] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.179160 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea18666ec324403b9f10a099688438b8" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:42.210405 855 leader_election.cc:343] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.210530 855 leader_election.cc:304] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:42.210932 2673 raft_consensus.cc:2743] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:42.212543 1177 tablet_bootstrap.cc:654] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.219138 2266 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" instance_seqno: 1736650342312772) as {username='slave'} at 127.0.104.2:51465; Asking this server to re-register.
I20250112 02:52:42.209434 2849 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250112 02:52:42.211867 2264 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f1e775296eb746db8b318f17d26820e6" instance_seqno: 1736650342780266) as {username='slave'} at 127.0.104.3:42047; Asking this server to re-register.
I20250112 02:52:42.235576 2445 raft_consensus.cc:491] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.235987 2445 raft_consensus.cc:513] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.236737 2445 leader_election.cc:290] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.237129 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1d9fae290ec4281839991f9d40b105c" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:42.237306 1051 raft_consensus.cc:2463] T b1d9fae290ec4281839991f9d40b105c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
W20250112 02:52:42.237663 713 leader_election.cc:343] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: b1d9fae290ec4281839991f9d40b105c
I20250112 02:52:42.237881 714 leader_election.cc:304] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:42.238297 2445 raft_consensus.cc:2798] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:42.238380 2445 raft_consensus.cc:491] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:42.238451 2445 raft_consensus.cc:3054] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.150058 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:42.241536 2445 raft_consensus.cc:513] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.242288 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:42.242542 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:42.242753 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:42.243031 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:42.243539 2445 leader_election.cc:290] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:42.243706 713 leader_election.cc:343] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: b1d9fae290ec4281839991f9d40b105c
I20250112 02:52:42.244784 2787 raft_consensus.cc:491] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.244895 2787 raft_consensus.cc:513] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.245168 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1d9fae290ec4281839991f9d40b105c" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:42.245323 1051 raft_consensus.cc:3054] T b1d9fae290ec4281839991f9d40b105c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.245400 2787 leader_election.cc:290] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:42.245869 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61eac6b572ff47c185768bee7490edf7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:42.246312 855 leader_election.cc:343] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.246735 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61eac6b572ff47c185768bee7490edf7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:42.247764 1051 raft_consensus.cc:2463] T b1d9fae290ec4281839991f9d40b105c P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:42.248206 714 leader_election.cc:304] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:42.248673 2840 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 436220c23fa948efb5b6ca15875d8b0e)
I20250112 02:52:42.248770 2451 raft_consensus.cc:2798] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:42.248875 2451 raft_consensus.cc:695] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:42.248886 2840 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:42.249099 2451 consensus_queue.cc:237] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.250699 2840 leader_election.cc:290] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 436220c23fa948efb5b6ca15875d8b0e (127.0.104.62:42341), dd25f03f25df49178a5c53c636ecf304 (127.0.104.60:39987)
W20250112 02:52:42.250941 854 leader_election.cc:343] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.251042 854 leader_election.cc:304] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:42.252684 2673 raft_consensus.cc:2743] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:42.261974 2853 tablet.cc:1721] T 3e67f0a0c065498f9a644220414415b0 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:42.261233 2855 tablet.cc:1721] T 3e67f0a0c065498f9a644220414415b0 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:42.262647 1173 tablet_bootstrap.cc:492] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:42.264719 1173 ts_tablet_manager.cc:1397] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.001s
I20250112 02:52:42.265336 1173 raft_consensus.cc:357] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.267566 1173 raft_consensus.cc:383] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.267640 1173 raft_consensus.cc:738] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.268841 1173 consensus_queue.cc:260] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.269372 1173 ts_tablet_manager.cc:1428] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250112 02:52:42.269591 1173 tablet_bootstrap.cc:492] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:42.270735 1173 tablet_bootstrap.cc:654] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.273751 966 heartbeater.cc:463] Registering TS with master...
I20250112 02:52:42.275933 2783 mvcc.cc:204] Tried to move back new op lower bound from 7113319861831610368 to 7113319836830162944. Current Snapshot: MvccSnapshot[applied={T|T < 7113319854219522048 or (T in {7113319854464749568,7113319857597136896,7113319859057225728,7113319859113418752,7113319859430752256,7113319859520294912,7113319859762089984,7113319859928539136,7113319859985371136,7113319860068495360,7113319860918476800,7113319860980400128})}]
I20250112 02:52:42.289610 2609 consensus_queue.cc:1035] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:42.291082 1102 heartbeater.cc:463] Registering TS with master...
I20250112 02:52:42.314244 2609 consensus_queue.cc:1035] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:42.320093 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 262) took 3164 ms. Trace:
I20250112 02:52:42.338474 2266 ts_manager.cc:194] Registered new tserver with Master: f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:42.338413 754 tablet_service.cc:1467] Processing CreateTablet for tablet d2028226498b4b41a95c7c836fe03b4d (DEFAULT_TABLE table=default.Table_ea6f20bc46ec487cae9e3abf6419c80c [id=34602f2c95734d478e38e1847de46aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.338829 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2028226498b4b41a95c7c836fe03b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.339058 895 tablet_service.cc:1467] Processing CreateTablet for tablet d2028226498b4b41a95c7c836fe03b4d (DEFAULT_TABLE table=default.Table_ea6f20bc46ec487cae9e3abf6419c80c [id=34602f2c95734d478e38e1847de46aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.339406 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2028226498b4b41a95c7c836fe03b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.341596 2865 tablet.cc:1721] T 3e67f0a0c065498f9a644220414415b0 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:42.342602 2264 ts_manager.cc:194] Registered new tserver with Master: fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.348327 2854 tablet.cc:1721] T bb4ef0220dde43569452379639759db5 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:42.352813 464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4a49a5756b143fcb767c3f41463f8d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 615 } } ignore_live_leader: false dest_uuid: "436220c23fa948efb5b6ca15875d8b0e" is_pre_election: true
I20250112 02:52:42.354386 2024 rpcz_store.cc:276] 0112 02:52:39.156059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.156299 (+ 240us) service_pool.cc:225] Handling call
0112 02:52:42.108825 (+2952526us) ts_tablet_manager.cc:1145] Deleting tablet 1dda52f012794493a102ed1abc718f5b
0112 02:52:42.108833 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.113341 (+ 4508us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.140185 (+ 26844us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.320072 (+179887us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2952410}
I20250112 02:52:42.354771 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 266) took 3198 ms. Trace:
I20250112 02:52:42.354885 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 1dda52f012794493a102ed1abc718f5b (table default.Table_8ab566cf9c8f45eba4dc9bb85264c823 [id=a622f7a973344a388b3275adbcce6234]) successfully deleted
I20250112 02:52:42.354861 2024 rpcz_store.cc:276] 0112 02:52:39.156416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.156675 (+ 259us) service_pool.cc:225] Handling call
0112 02:52:42.354678 (+3198003us) ts_tablet_manager.cc:1145] Deleting tablet 7a18132549374218893f35a4e4226b72
0112 02:52:42.354688 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.354756 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3197859}
I20250112 02:52:42.355171 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 265) took 3198 ms. Trace:
I20250112 02:52:42.355067 754 tablet_service.cc:1467] Processing CreateTablet for tablet f5c671e327a148ecb1d08a983448946e (DEFAULT_TABLE table=default.Table_80b349ce76644bcabf626e044231436a [id=ba3ca203a209474184c7763dabea3f5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.355245 2024 rpcz_store.cc:276] 0112 02:52:39.156297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.157082 (+ 785us) service_pool.cc:225] Handling call
0112 02:52:42.355078 (+3197996us) ts_tablet_manager.cc:1145] Deleting tablet 058a9ca089cd4967887b0a1127c7d03f
0112 02:52:42.355088 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.355157 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3197863}
W20250112 02:52:42.355309 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 7a18132549374218893f35a4e4226b72 because the tablet was not found. No further retry: Not found: Tablet not found: 7a18132549374218893f35a4e4226b72
I20250112 02:52:42.355520 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 271) took 3198 ms. Trace:
I20250112 02:52:42.355640 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c671e327a148ecb1d08a983448946e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.355579 2024 rpcz_store.cc:276] 0112 02:52:39.156853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.157243 (+ 390us) service_pool.cc:225] Handling call
0112 02:52:42.355426 (+3198183us) ts_tablet_manager.cc:1145] Deleting tablet 9d5b3df88a0c45dba879b3d506710614
0112 02:52:42.355434 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.355505 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3198048}
I20250112 02:52:42.355866 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 256) took 3200 ms. Trace:
I20250112 02:52:42.355947 2024 rpcz_store.cc:276] 0112 02:52:39.155518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.157244 (+ 1726us) service_pool.cc:225] Handling call
0112 02:52:42.355796 (+3198552us) ts_tablet_manager.cc:1145] Deleting tablet 6913b0ac9ceb4dbc9cdf591f159134f7
0112 02:52:42.355804 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.355853 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3198404}
W20250112 02:52:42.356081 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 058a9ca089cd4967887b0a1127c7d03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 058a9ca089cd4967887b0a1127c7d03f already in progress: creating tablet
I20250112 02:52:42.356200 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 259) took 3200 ms. Trace:
W20250112 02:52:42.356302 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 9d5b3df88a0c45dba879b3d506710614 because the tablet was not found. No further retry: Not found: Tablet not found: 9d5b3df88a0c45dba879b3d506710614
I20250112 02:52:42.356267 2024 rpcz_store.cc:276] 0112 02:52:39.155792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.157423 (+ 1631us) service_pool.cc:225] Handling call
0112 02:52:42.356134 (+3198711us) ts_tablet_manager.cc:1145] Deleting tablet a85300a9aa1c4c50a3957a004bea7933
0112 02:52:42.356142 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.356188 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3198587}
I20250112 02:52:42.356528 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 261) took 3200 ms. Trace:
I20250112 02:52:42.356595 2024 rpcz_store.cc:276] 0112 02:52:39.155971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.157576 (+ 1605us) service_pool.cc:225] Handling call
0112 02:52:42.356453 (+3198877us) ts_tablet_manager.cc:1145] Deleting tablet 1b184806b4884cd3b0a002905a1e57af
0112 02:52:42.356463 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.356514 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3198742}
W20250112 02:52:42.356745 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 6913b0ac9ceb4dbc9cdf591f159134f7 because the tablet was not found. No further retry: Not found: Tablet not found: 6913b0ac9ceb4dbc9cdf591f159134f7
I20250112 02:52:42.356794 2264 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" instance_seqno: 1736650341878407) as {username='slave'} at 127.0.104.1:44327; Asking this server to re-register.
I20250112 02:52:42.356853 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 264) took 3200 ms. Trace:
W20250112 02:52:42.356961 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet a85300a9aa1c4c50a3957a004bea7933 because the tablet was not found. No further retry: Not found: Tablet not found: a85300a9aa1c4c50a3957a004bea7933
I20250112 02:52:42.356910 2024 rpcz_store.cc:276] 0112 02:52:39.156220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.157990 (+ 1770us) service_pool.cc:225] Handling call
0112 02:52:42.356789 (+3198799us) ts_tablet_manager.cc:1145] Deleting tablet c8a7281c0c09442ebd0cc13b8496c86e
0112 02:52:42.356798 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.356842 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3198673}
I20250112 02:52:42.357136 2024 tablet_replica.cc:331] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:42.357303 2024 raft_consensus.cc:2238] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:42.357564 2024 pending_rounds.cc:62] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6: Trying to abort 1 pending ops.
I20250112 02:52:42.357615 2024 pending_rounds.cc:69] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319867460427776 op_type: NO_OP noop_request { }
I20250112 02:52:42.357703 2024 raft_consensus.cc:2883] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:42.357771 2024 raft_consensus.cc:2267] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:42.358286 2024 ts_tablet_manager.cc:1905] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:42.358353 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 1b184806b4884cd3b0a002905a1e57af because the tablet was not found. No further retry: Not found: Tablet not found: 1b184806b4884cd3b0a002905a1e57af
W20250112 02:52:42.358551 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet c8a7281c0c09442ebd0cc13b8496c86e because the tablet was not found. No further retry: Not found: Tablet not found: c8a7281c0c09442ebd0cc13b8496c86e
I20250112 02:52:42.354386 895 tablet_service.cc:1467] Processing CreateTablet for tablet f5c671e327a148ecb1d08a983448946e (DEFAULT_TABLE table=default.Table_80b349ce76644bcabf626e044231436a [id=ba3ca203a209474184c7763dabea3f5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.358958 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c671e327a148ecb1d08a983448946e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.321027 1173 tablet_bootstrap.cc:492] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:42.322256 2853 tablet.cc:1721] T bb4ef0220dde43569452379639759db5 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:42.340099 1017 tablet_service.cc:1467] Processing CreateTablet for tablet d2028226498b4b41a95c7c836fe03b4d (DEFAULT_TABLE table=default.Table_ea6f20bc46ec487cae9e3abf6419c80c [id=34602f2c95734d478e38e1847de46aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.359781 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2028226498b4b41a95c7c836fe03b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.362540 827 heartbeater.cc:463] Registering TS with master...
I20250112 02:52:42.359077 2855 tablet.cc:1721] T bb4ef0220dde43569452379639759db5 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:42.353271 1031 tablet_service.cc:1467] Processing CreateTablet for tablet f5c671e327a148ecb1d08a983448946e (DEFAULT_TABLE table=default.Table_80b349ce76644bcabf626e044231436a [id=ba3ca203a209474184c7763dabea3f5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.363708 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c671e327a148ecb1d08a983448946e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.353382 2609 raft_consensus.cc:491] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.368325 2609 raft_consensus.cc:513] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.369313 2609 leader_election.cc:290] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.369333 2264 ts_manager.cc:194] Registered new tserver with Master: 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:42.370904 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6650515ea540a8bbf21e2f88363360" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:42.371071 917 raft_consensus.cc:2463] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:42.371388 989 leader_election.cc:304] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:42.372259 2609 raft_consensus.cc:2798] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:42.372474 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6650515ea540a8bbf21e2f88363360" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:42.372741 2609 raft_consensus.cc:491] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:42.372834 2609 raft_consensus.cc:3054] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:42.372949 991 leader_election.cc:343] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.373251 1173 ts_tablet_manager.cc:1397] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.104s user 0.002s sys 0.000s
I20250112 02:52:42.373886 1173 raft_consensus.cc:357] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.374091 1173 raft_consensus.cc:383] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.374176 1173 raft_consensus.cc:738] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.374653 1173 consensus_queue.cc:260] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.375149 2609 raft_consensus.cc:513] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.375303 1173 ts_tablet_manager.cc:1428] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:42.375619 1173 tablet_bootstrap.cc:492] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:42.376119 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6650515ea540a8bbf21e2f88363360" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:42.376585 1181 tablet_bootstrap.cc:492] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
W20250112 02:52:42.376581 991 leader_election.cc:343] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.377015 2609 leader_election.cc:290] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.377328 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6650515ea540a8bbf21e2f88363360" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:42.377462 917 raft_consensus.cc:3054] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.378824 743 tablet_service.cc:1467] Processing CreateTablet for tablet a9e31bccdd30492ab73dc0990a61a133 (DEFAULT_TABLE table=default.Table_e87ea43667cd4f4eae136a1614aa9a67 [id=15cdcd7b7f514f74b76727a4ef1d110f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.379161 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9e31bccdd30492ab73dc0990a61a133. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.380769 917 raft_consensus.cc:2463] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:42.376798 1181 ts_tablet_manager.cc:1397] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.269s user 0.007s sys 0.002s
I20250112 02:52:42.381202 989 leader_election.cc:304] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:42.381660 1181 raft_consensus.cc:357] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.381794 2745 raft_consensus.cc:2798] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:42.381920 1181 raft_consensus.cc:383] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.381922 2745 raft_consensus.cc:695] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:42.382036 1181 raft_consensus.cc:738] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.382140 2745 consensus_queue.cc:237] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.382292 1181 consensus_queue.cc:260] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.382866 1181 ts_tablet_manager.cc:1428] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:42.383155 1181 tablet_bootstrap.cc:492] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
W20250112 02:52:42.384008 991 consensus_peers.cc:487] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 795ac7eaf81d48bb926865d9ebb388b6. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:42.384433 2673 raft_consensus.cc:491] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.384541 2673 raft_consensus.cc:513] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.384634 1031 tablet_service.cc:1467] Processing CreateTablet for tablet e571910bf3564e6b9f58a1e0a35d784d (DEFAULT_TABLE table=default.Table_8d89d19f0fb24628b94e2c615bf9f787 [id=8c1e8a2de51844d198d18133b47c4837]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.384927 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e571910bf3564e6b9f58a1e0a35d784d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.385089 2673 leader_election.cc:290] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:42.385571 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09900b42e8a443e680cc35c96a1dbbb7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:42.385840 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09900b42e8a443e680cc35c96a1dbbb7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:42.385959 855 leader_election.cc:343] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:42.386325 854 leader_election.cc:343] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.386413 854 leader_election.cc:304] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:42.386689 2673 raft_consensus.cc:2743] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:42.389855 895 tablet_service.cc:1467] Processing CreateTablet for tablet e571910bf3564e6b9f58a1e0a35d784d (DEFAULT_TABLE table=default.Table_8d89d19f0fb24628b94e2c615bf9f787 [id=8c1e8a2de51844d198d18133b47c4837]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.390228 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e571910bf3564e6b9f58a1e0a35d784d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.390252 754 tablet_service.cc:1467] Processing CreateTablet for tablet e571910bf3564e6b9f58a1e0a35d784d (DEFAULT_TABLE table=default.Table_8d89d19f0fb24628b94e2c615bf9f787 [id=8c1e8a2de51844d198d18133b47c4837]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.390614 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e571910bf3564e6b9f58a1e0a35d784d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.394472 1017 tablet_service.cc:1467] Processing CreateTablet for tablet a9e31bccdd30492ab73dc0990a61a133 (DEFAULT_TABLE table=default.Table_e87ea43667cd4f4eae136a1614aa9a67 [id=15cdcd7b7f514f74b76727a4ef1d110f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.394826 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9e31bccdd30492ab73dc0990a61a133. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.376978 1173 tablet_bootstrap.cc:654] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.414701 743 tablet_service.cc:1467] Processing CreateTablet for tablet 138c0a1bb2f04edbb21a32a5b8abddf2 (DEFAULT_TABLE table=default.Table_c80abe729cbf4e339b28b3ad6b3ba247 [id=a0d7b1b214f34ecbafaf0693101bc958]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.415112 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 138c0a1bb2f04edbb21a32a5b8abddf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.415079 897 tablet_service.cc:1467] Processing CreateTablet for tablet a9e31bccdd30492ab73dc0990a61a133 (DEFAULT_TABLE table=default.Table_e87ea43667cd4f4eae136a1614aa9a67 [id=15cdcd7b7f514f74b76727a4ef1d110f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.420852 1181 tablet_bootstrap.cc:654] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.423121 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9e31bccdd30492ab73dc0990a61a133. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.423986 1173 tablet_bootstrap.cc:492] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:42.424170 1173 ts_tablet_manager.cc:1397] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20250112 02:52:42.424494 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet bb4ef0220dde43569452379639759db5 (table default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed]). Expected version 2 got 0
I20250112 02:52:42.424917 1173 raft_consensus.cc:357] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.425105 1173 raft_consensus.cc:383] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.425316 1173 raft_consensus.cc:738] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.425592 1173 consensus_queue.cc:260] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.425526 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 138c0a1bb2f04edbb21a32a5b8abddf2 (DEFAULT_TABLE table=default.Table_c80abe729cbf4e339b28b3ad6b3ba247 [id=a0d7b1b214f34ecbafaf0693101bc958]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.426182 1173 ts_tablet_manager.cc:1428] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:42.426868 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 818cd6362b94473ea06ed68fa5ffd1bc (DEFAULT_TABLE table=default.Table_178713b464bb4f16970851646a710574 [id=d3f31247f0e848f8b2dfa18f3437ca63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.426952 1173 tablet_bootstrap.cc:492] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:42.427906 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf8a907f0af4e24819d1f00e40885ee (DEFAULT_TABLE table=default.Table_6e552be8f8b7468f84f7675b3342125e [id=c7785d8276dc44608e5a3e0657bbce39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.428395 1173 tablet_bootstrap.cc:654] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.428661 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 818cd6362b94473ea06ed68fa5ffd1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.429582 899 tablet_service.cc:1467] Processing CreateTablet for tablet 818cd6362b94473ea06ed68fa5ffd1bc (DEFAULT_TABLE table=default.Table_178713b464bb4f16970851646a710574 [id=d3f31247f0e848f8b2dfa18f3437ca63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.429690 897 tablet_service.cc:1467] Processing CreateTablet for tablet 138c0a1bb2f04edbb21a32a5b8abddf2 (DEFAULT_TABLE table=default.Table_c80abe729cbf4e339b28b3ad6b3ba247 [id=a0d7b1b214f34ecbafaf0693101bc958]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.429888 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 818cd6362b94473ea06ed68fa5ffd1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.431572 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 138c0a1bb2f04edbb21a32a5b8abddf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.431849 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bf8a907f0af4e24819d1f00e40885ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.432446 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 138c0a1bb2f04edbb21a32a5b8abddf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.433550 890 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf8a907f0af4e24819d1f00e40885ee (DEFAULT_TABLE table=default.Table_6e552be8f8b7468f84f7675b3342125e [id=c7785d8276dc44608e5a3e0657bbce39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.433825 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bf8a907f0af4e24819d1f00e40885ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.436223 1181 tablet_bootstrap.cc:492] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:42.436336 1181 ts_tablet_manager.cc:1397] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.001s
I20250112 02:52:42.437047 1181 raft_consensus.cc:357] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.437219 1181 raft_consensus.cc:383] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.437278 1181 raft_consensus.cc:738] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.437577 1181 consensus_queue.cc:260] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.438270 1181 ts_tablet_manager.cc:1428] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:42.438529 1181 tablet_bootstrap.cc:492] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:42.439239 2024 ts_tablet_manager.cc:1918] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:42.439343 2024 log.cc:1198] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/b8daf4ef3593410eae87672402292be6
I20250112 02:52:42.444680 1181 tablet_bootstrap.cc:654] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.445699 756 tablet_service.cc:1467] Processing CreateTablet for tablet 818cd6362b94473ea06ed68fa5ffd1bc (DEFAULT_TABLE table=default.Table_178713b464bb4f16970851646a710574 [id=d3f31247f0e848f8b2dfa18f3437ca63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.446070 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 818cd6362b94473ea06ed68fa5ffd1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.450618 1177 tablet_bootstrap.cc:492] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:42.450735 1177 ts_tablet_manager.cc:1397] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.306s user 0.003s sys 0.001s
I20250112 02:52:42.451074 760 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf8a907f0af4e24819d1f00e40885ee (DEFAULT_TABLE table=default.Table_6e552be8f8b7468f84f7675b3342125e [id=c7785d8276dc44608e5a3e0657bbce39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.456104 1177 raft_consensus.cc:357] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.456271 1177 raft_consensus.cc:383] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.456311 1177 raft_consensus.cc:738] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.457893 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bf8a907f0af4e24819d1f00e40885ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.459781 744 tablet_service.cc:1467] Processing CreateTablet for tablet 97c7ce72d02049089c653684c53cb53c (DEFAULT_TABLE table=default.Table_f0fb142b9aab490bb067d3cf64dc47b1 [id=1991c0821ef4478eb0eb9282ab772c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.460062 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97c7ce72d02049089c653684c53cb53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.467600 1177 consensus_queue.cc:260] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.468164 1177 ts_tablet_manager.cc:1428] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250112 02:52:42.468426 1177 tablet_bootstrap.cc:492] T 7ad9c47fcb534619949971796b60eba2 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:42.470062 1177 tablet_bootstrap.cc:654] T 7ad9c47fcb534619949971796b60eba2 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.439716 2024 ts_tablet_manager.cc:1939] T b8daf4ef3593410eae87672402292be6 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:42.472218 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 267) took 3315 ms. Trace:
I20250112 02:52:42.473183 2024 rpcz_store.cc:276] 0112 02:52:39.156497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.159757 (+ 3260us) service_pool.cc:225] Handling call
0112 02:52:42.357095 (+3197338us) ts_tablet_manager.cc:1145] Deleting tablet b8daf4ef3593410eae87672402292be6
0112 02:52:42.357104 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.419129 (+ 62025us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.439162 (+ 20033us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.472199 (+ 33037us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3197153}
I20250112 02:52:42.473539 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 254) took 3318 ms. Trace:
I20250112 02:52:42.473613 2024 rpcz_store.cc:276] 0112 02:52:39.155337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.164055 (+ 8718us) service_pool.cc:225] Handling call
0112 02:52:42.473442 (+3309387us) ts_tablet_manager.cc:1145] Deleting tablet 87759e8b99b04e628badc8f60cb38bb3
0112 02:52:42.473462 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.473524 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3309254}
I20250112 02:52:42.473871 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 255) took 3318 ms. Trace:
I20250112 02:52:42.473937 2024 rpcz_store.cc:276] 0112 02:52:39.155414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.164215 (+ 8801us) service_pool.cc:225] Handling call
0112 02:52:42.473807 (+3309592us) ts_tablet_manager.cc:1145] Deleting tablet 1c639a2bcd4740e4be7b2b162004669a
0112 02:52:42.473814 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.473859 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3309466}
I20250112 02:52:42.474181 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 253) took 3318 ms. Trace:
I20250112 02:52:42.474251 2228 consensus_queue.cc:1035] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:42.475651 2228 raft_consensus.cc:491] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.475759 2228 raft_consensus.cc:513] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.476279 2228 leader_election.cc:290] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.478127 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5b3c712c0f448ea9b14ab93dd731ab" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:42.474258 2024 rpcz_store.cc:276] 0112 02:52:39.155269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.163980 (+ 8711us) service_pool.cc:225] Handling call
0112 02:52:42.474117 (+3310137us) ts_tablet_manager.cc:1145] Deleting tablet 61593969d2234ce3a6089027b067d922
0112 02:52:42.474124 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.474169 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3309682}
W20250112 02:52:42.478428 989 leader_election.cc:343] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: bd5b3c712c0f448ea9b14ab93dd731ab
I20250112 02:52:42.478487 2024 tablet_replica.cc:331] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
W20250112 02:52:42.478554 991 leader_election.cc:343] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.478643 991 leader_election.cc:304] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:42.479081 2228 raft_consensus.cc:2743] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:42.478648 2024 raft_consensus.cc:2238] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:42.483815 2024 pending_rounds.cc:62] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6: Trying to abort 1 pending ops.
I20250112 02:52:42.483876 2024 pending_rounds.cc:69] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319874163109888 op_type: NO_OP noop_request { }
I20250112 02:52:42.483970 2024 raft_consensus.cc:2883] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:42.484045 2024 raft_consensus.cc:2267] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:42.484560 2024 ts_tablet_manager.cc:1905] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:42.484714 2228 consensus_queue.cc:1035] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:42.486927 1393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4a49a5756b143fcb767c3f41463f8d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 615 } } ignore_live_leader: false dest_uuid: "dd25f03f25df49178a5c53c636ecf304" is_pre_election: true
I20250112 02:52:42.487793 2246 leader_election.cc:304] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4a49a5756b143fcb767c3f41463f8d9; no voters: 436220c23fa948efb5b6ca15875d8b0e, dd25f03f25df49178a5c53c636ecf304
I20250112 02:52:42.488461 771 raft_consensus.cc:3054] T 2052b4536f9745f0a785f330d706c1cf P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.490579 2840 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250112 02:52:42.491858 2228 consensus_queue.cc:1035] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:42.495806 760 tablet_service.cc:1467] Processing CreateTablet for tablet 99543199a2ec40f8ab170a52fb663067 (DEFAULT_TABLE table=default.Table_99896ca24c1a4eb1a31b036b021a8621 [id=f06b24bb413f47afa1b398562b1e183f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.496399 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99543199a2ec40f8ab170a52fb663067. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.477104 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet b8daf4ef3593410eae87672402292be6 (table default.Table_c0aaadb5e69e46e8983f4d289a44afe2 [id=08299ab225a94e37a67722107a5ff4ce]) successfully deleted
W20250112 02:52:42.497730 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 87759e8b99b04e628badc8f60cb38bb3 because the tablet was not found. No further retry: Not found: Tablet not found: 87759e8b99b04e628badc8f60cb38bb3
W20250112 02:52:42.497893 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 1c639a2bcd4740e4be7b2b162004669a because the tablet was not found. No further retry: Not found: Tablet not found: 1c639a2bcd4740e4be7b2b162004669a
W20250112 02:52:42.498042 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 61593969d2234ce3a6089027b067d922 because the tablet was not found. No further retry: Not found: Tablet not found: 61593969d2234ce3a6089027b067d922
W20250112 02:52:42.482515 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 7e1954e7859f422f9690fe4a9384a8bb (table default.Table_829049fe22b74e62a87dd1bed65fb3fe [id=2e892eba68234ffca0be4ef3e49cf309], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7e1954e7859f422f9690fe4a9384a8bb consensus state has no leader
I20250112 02:52:42.499059 899 tablet_service.cc:1467] Processing CreateTablet for tablet 99543199a2ec40f8ab170a52fb663067 (DEFAULT_TABLE table=default.Table_99896ca24c1a4eb1a31b036b021a8621 [id=f06b24bb413f47afa1b398562b1e183f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.499208 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 99543199a2ec40f8ab170a52fb663067 (DEFAULT_TABLE table=default.Table_99896ca24c1a4eb1a31b036b021a8621 [id=f06b24bb413f47afa1b398562b1e183f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.499454 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99543199a2ec40f8ab170a52fb663067. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.504618 2228 raft_consensus.cc:491] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.505965 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 97c7ce72d02049089c653684c53cb53c (DEFAULT_TABLE table=default.Table_f0fb142b9aab490bb067d3cf64dc47b1 [id=1991c0821ef4478eb0eb9282ab772c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.506613 2673 consensus_queue.cc:1035] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:42.507086 1173 tablet_bootstrap.cc:492] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:42.507113 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:42.507192 1173 ts_tablet_manager.cc:1397] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.080s user 0.006s sys 0.002s
I20250112 02:52:42.507292 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 46d1beac4410422bbf1e912d35999f3d (table default.Table_66ecd84e49d64f50b63b520f74b8e062 [id=3a0f5ac3329d4b60befce41c91f5ae5c]). Expected version 1 got 0
I20250112 02:52:42.507752 895 tablet_service.cc:1467] Processing CreateTablet for tablet 97c7ce72d02049089c653684c53cb53c (DEFAULT_TABLE table=default.Table_f0fb142b9aab490bb067d3cf64dc47b1 [id=1991c0821ef4478eb0eb9282ab772c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:42.507817 1173 raft_consensus.cc:357] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.507858 2228 raft_consensus.cc:513] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.508040 1173 raft_consensus.cc:383] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.508103 1173 raft_consensus.cc:738] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.508473 2228 leader_election.cc:290] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.508440 1173 consensus_queue.cc:260] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.509737 1173 ts_tablet_manager.cc:1428] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:42.510726 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99543199a2ec40f8ab170a52fb663067. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.512023 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b124c15736084e6d8f69b4fb7c41cf39" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:42.512639 991 leader_election.cc:343] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.512862 2787 raft_consensus.cc:491] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.513033 2787 raft_consensus.cc:513] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.513634 2787 leader_election.cc:290] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:42.514582 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3499374810fb412e938325434c35618d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:42.515033 854 leader_election.cc:343] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.515288 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3499374810fb412e938325434c35618d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:42.515753 855 leader_election.cc:343] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.515846 855 leader_election.cc:304] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:42.516181 2673 raft_consensus.cc:2743] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:42.516376 2853 tablet.cc:1721] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:42.516710 2451 raft_consensus.cc:491] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.516811 2451 raft_consensus.cc:513] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.517380 2451 leader_election.cc:290] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.517666 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ded8bcdda914ea6bc4f9bb92d413dee" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:42.517922 2865 tablet.cc:1721] T 47a0d939fdb94590a86389f1f4693e9a P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:52:42.518050 714 leader_election.cc:343] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.519145 2228 raft_consensus.cc:491] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.519300 2228 raft_consensus.cc:513] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.519989 2228 leader_election.cc:290] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.527374 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f3be6bcbe8a45f0a8076b4842d90dc2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:42.528072 991 leader_election.cc:343] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.528420 2228 consensus_queue.cc:1035] T e4b7c2efd7ea402e8d19038f96b9675c P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:42.531817 2673 consensus_queue.cc:1035] T fea2ef4e9a63491db850bb5393938017 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:42.537984 1177 tablet_bootstrap.cc:492] T 7ad9c47fcb534619949971796b60eba2 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:42.541013 2024 ts_tablet_manager.cc:1918] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:42.541102 2024 log.cc:1198] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/4348fe4c32704bf7b3a8d21ff3b8ac1b
I20250112 02:52:42.541404 2024 ts_tablet_manager.cc:1939] T 4348fe4c32704bf7b3a8d21ff3b8ac1b P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:42.547976 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97c7ce72d02049089c653684c53cb53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.549235 1177 ts_tablet_manager.cc:1397] T 7ad9c47fcb534619949971796b60eba2 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.081s user 0.005s sys 0.001s
I20250112 02:52:42.550518 1177 raft_consensus.cc:357] T 7ad9c47fcb534619949971796b60eba2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.554771 1177 raft_consensus.cc:383] T 7ad9c47fcb534619949971796b60eba2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.554993 1177 raft_consensus.cc:738] T 7ad9c47fcb534619949971796b60eba2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.555404 1177 consensus_queue.cc:260] T 7ad9c47fcb534619949971796b60eba2 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.556586 1177 ts_tablet_manager.cc:1428] T 7ad9c47fcb534619949971796b60eba2 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250112 02:52:42.558833 1173 tablet_bootstrap.cc:492] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:42.559159 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 252) took 3404 ms. Trace:
I20250112 02:52:42.559228 2024 rpcz_store.cc:276] 0112 02:52:39.155178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.164456 (+ 9278us) service_pool.cc:225] Handling call
0112 02:52:42.478450 (+3313994us) ts_tablet_manager.cc:1145] Deleting tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b
0112 02:52:42.478458 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.510680 (+ 32222us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.540933 (+ 30253us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.559142 (+ 18209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3313886}
I20250112 02:52:42.559566 2024 tablet_replica.cc:331] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:42.559756 2024 raft_consensus.cc:2238] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:42.560066 2024 raft_consensus.cc:2267] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:42.560387 1173 tablet_bootstrap.cc:654] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.560530 2024 ts_tablet_manager.cc:1905] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:42.561753 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 4348fe4c32704bf7b3a8d21ff3b8ac1b (table default.Table_1d107b48a09e41d6bb3401b01a21ee78 [id=68bab352836142c39c109e8e02553802]) successfully deleted
I20250112 02:52:42.569671 1173 tablet_bootstrap.cc:492] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:42.569941 1173 ts_tablet_manager.cc:1397] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20250112 02:52:42.570506 1173 raft_consensus.cc:357] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.570690 1173 raft_consensus.cc:383] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.570755 1173 raft_consensus.cc:738] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.569965 2024 ts_tablet_manager.cc:1918] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 02:52:42.570935 2024 log.cc:1198] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/f31b777e7e0f43a28839d8c48c6a6590
I20250112 02:52:42.570962 1173 consensus_queue.cc:260] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.571247 2024 ts_tablet_manager.cc:1939] T f31b777e7e0f43a28839d8c48c6a6590 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:42.571614 1173 ts_tablet_manager.cc:1428] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:42.571846 1173 tablet_bootstrap.cc:492] T 41d5799b5b36487da42dc43b1c70ae26 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:42.571887 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 272) took 3237 ms. Trace:
I20250112 02:52:42.573184 2228 consensus_queue.cc:1035] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:42.573348 1173 tablet_bootstrap.cc:654] T 41d5799b5b36487da42dc43b1c70ae26 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.575927 1181 tablet_bootstrap.cc:492] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:42.576090 1181 ts_tablet_manager.cc:1397] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.138s user 0.001s sys 0.005s
I20250112 02:52:42.577698 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ded8bcdda914ea6bc4f9bb92d413dee" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:42.577873 916 raft_consensus.cc:2463] T 9ded8bcdda914ea6bc4f9bb92d413dee P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:42.578061 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f3be6bcbe8a45f0a8076b4842d90dc2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:42.578212 912 raft_consensus.cc:2463] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
W20250112 02:52:42.579250 989 leader_election.cc:343] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: b124c15736084e6d8f69b4fb7c41cf39
I20250112 02:52:42.579236 1181 raft_consensus.cc:357] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.579345 989 leader_election.cc:304] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:42.579413 1181 raft_consensus.cc:383] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.579459 1181 raft_consensus.cc:738] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.579766 2609 raft_consensus.cc:2743] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:42.579896 989 leader_election.cc:304] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:42.580029 2609 consensus_queue.cc:1035] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:42.580359 2745 raft_consensus.cc:2798] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:42.580427 2745 raft_consensus.cc:491] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:42.580484 2745 raft_consensus.cc:3054] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.571974 2024 rpcz_store.cc:276] 0112 02:52:39.334856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:39.335532 (+ 676us) service_pool.cc:225] Handling call
0112 02:52:42.559520 (+3223988us) ts_tablet_manager.cc:1145] Deleting tablet f31b777e7e0f43a28839d8c48c6a6590
0112 02:52:42.559532 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.563159 (+ 3627us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.569896 (+ 6737us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.571871 (+ 1975us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3219437}
I20250112 02:52:42.581069 2024 tablet_replica.cc:331] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:42.581239 2024 raft_consensus.cc:2238] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:42.581498 2024 raft_consensus.cc:2267] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:42.581941 2024 ts_tablet_manager.cc:1905] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:42.582701 2745 raft_consensus.cc:513] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.583243 2745 leader_election.cc:290] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.584367 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet f31b777e7e0f43a28839d8c48c6a6590 (table default.Table_41b7056b6e9240928d0af52aec2e9338 [id=6402abce8d1d45c182cc989490c6a5f0]) successfully deleted
I20250112 02:52:42.584988 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f3be6bcbe8a45f0a8076b4842d90dc2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:42.585175 919 raft_consensus.cc:3054] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.585611 2876 raft_consensus.cc:491] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.585752 2876 raft_consensus.cc:513] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.585863 965 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:42.586337 2876 leader_election.cc:290] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:42.589951 2855 tablet.cc:1721] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:42.591661 919 raft_consensus.cc:2463] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:42.592032 989 leader_election.cc:304] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:42.592255 2228 raft_consensus.cc:2798] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:42.592368 2228 raft_consensus.cc:695] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:42.592572 2228 consensus_queue.cc:237] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.594338 2673 raft_consensus.cc:491] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.594455 2673 raft_consensus.cc:513] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.595016 2673 leader_election.cc:290] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:42.596896 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d47f6d663934b25a0200ee9fa81165d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:42.597340 854 leader_election.cc:343] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.599879 1181 consensus_queue.cc:260] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.600222 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37862023095644bab2403576e8cb8c58" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:42.600437 1049 raft_consensus.cc:2463] T 37862023095644bab2403576e8cb8c58 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:42.600812 854 leader_election.cc:304] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:42.578236 713 leader_election.cc:304] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:42.589766 826 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:42.606791 2179 raft_consensus.cc:2798] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:42.606886 2179 raft_consensus.cc:491] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:42.606966 2179 raft_consensus.cc:3054] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.590761 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37862023095644bab2403576e8cb8c58" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:42.607555 776 raft_consensus.cc:2463] T 37862023095644bab2403576e8cb8c58 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:42.595691 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f3be6bcbe8a45f0a8076b4842d90dc2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:42.595969 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d47f6d663934b25a0200ee9fa81165d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:42.608659 991 leader_election.cc:343] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.612712 2673 raft_consensus.cc:491] T 54d3664071b146a3ac7c6885f9bda789 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.612838 2673 raft_consensus.cc:513] T 54d3664071b146a3ac7c6885f9bda789 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.613366 2673 leader_election.cc:290] T 54d3664071b146a3ac7c6885f9bda789 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:42.614073 2179 raft_consensus.cc:513] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.614949 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ded8bcdda914ea6bc4f9bb92d413dee" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:42.615073 2179 leader_election.cc:290] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.615265 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ded8bcdda914ea6bc4f9bb92d413dee" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
W20250112 02:52:42.615352 714 leader_election.cc:343] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.615418 917 raft_consensus.cc:3054] T 9ded8bcdda914ea6bc4f9bb92d413dee P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.617913 917 raft_consensus.cc:2463] T 9ded8bcdda914ea6bc4f9bb92d413dee P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:42.618233 713 leader_election.cc:304] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:42.618985 1181 ts_tablet_manager.cc:1428] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.043s user 0.001s sys 0.000s
I20250112 02:52:42.619248 1181 tablet_bootstrap.cc:492] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:42.619550 2179 raft_consensus.cc:2798] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:42.619735 2179 raft_consensus.cc:695] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:42.619925 2179 consensus_queue.cc:237] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.620666 1181 tablet_bootstrap.cc:654] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.624661 2024 ts_tablet_manager.cc:1918] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:42.624769 2024 log.cc:1198] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/f924459d1150472aafb325a11825db23
I20250112 02:52:42.625140 2024 ts_tablet_manager.cc:1939] T f924459d1150472aafb325a11825db23 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:42.625932 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 274) took 2474 ms. Trace:
I20250112 02:52:42.626026 2024 rpcz_store.cc:276] 0112 02:52:40.151925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.152092 (+ 167us) service_pool.cc:225] Handling call
0112 02:52:42.581032 (+2428940us) ts_tablet_manager.cc:1145] Deleting tablet f924459d1150472aafb325a11825db23
0112 02:52:42.581040 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.621701 (+ 40661us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.624591 (+ 2890us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.625915 (+ 1324us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2421486}
I20250112 02:52:42.626413 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 275) took 2474 ms. Trace:
I20250112 02:52:42.626435 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet f924459d1150472aafb325a11825db23 (table default.Table_48ca697307d74d89aa61ef5432217ca9 [id=67a1bb5c1e254f0f9f7f3b7d1c2299d1]) successfully deleted
I20250112 02:52:42.626484 2024 rpcz_store.cc:276] 0112 02:52:40.152039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.152250 (+ 211us) service_pool.cc:225] Handling call
0112 02:52:42.626309 (+2474059us) ts_tablet_manager.cc:1145] Deleting tablet f3771166b77e4f90aaa116ab1a9dc299
0112 02:52:42.626319 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.626401 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2466707}
I20250112 02:52:42.626700 2024 tablet_replica.cc:331] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
W20250112 02:52:42.626820 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f3771166b77e4f90aaa116ab1a9dc299 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3771166b77e4f90aaa116ab1a9dc299 already in progress: creating tablet
I20250112 02:52:42.626884 2024 raft_consensus.cc:2238] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:42.627014 2024 raft_consensus.cc:2267] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:42.638120 2024 ts_tablet_manager.cc:1905] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:42.642473 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97c7ce72d02049089c653684c53cb53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:42.643770 1177 tablet_bootstrap.cc:492] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:42.645387 1177 tablet_bootstrap.cc:654] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.646607 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:42.646886 2024 ts_tablet_manager.cc:1918] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:42.646984 2024 log.cc:1198] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/3e378a6a9b4a455c9f5e1d1ec0fd6ca0
I20250112 02:52:42.647328 2024 ts_tablet_manager.cc:1939] T 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:42.647643 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:42.651257 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:42.652369 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 281) took 2303 ms. Trace:
I20250112 02:52:42.652451 2024 rpcz_store.cc:276] 0112 02:52:40.348589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.348636 (+ 47us) service_pool.cc:225] Handling call
0112 02:52:42.626664 (+2278028us) ts_tablet_manager.cc:1145] Deleting tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0
0112 02:52:42.626672 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.641349 (+ 14677us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.646810 (+ 5461us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.652350 (+ 5540us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2277849}
I20250112 02:52:42.652765 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 293) took 1728 ms. Trace:
I20250112 02:52:42.652827 2024 rpcz_store.cc:276] 0112 02:52:40.924290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:40.937638 (+ 13348us) service_pool.cc:225] Handling call
0112 02:52:42.652687 (+1715049us) ts_tablet_manager.cc:1145] Deleting tablet 3b9280db29e54d41af1bed2fa5aa5535
0112 02:52:42.652694 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.652752 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1714852}
I20250112 02:52:42.653116 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 294) took 1608 ms. Trace:
I20250112 02:52:42.653244 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]) successfully deleted
I20250112 02:52:42.653192 2024 rpcz_store.cc:276] 0112 02:52:41.044332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.053609 (+ 9277us) service_pool.cc:225] Handling call
0112 02:52:42.653014 (+1599405us) ts_tablet_manager.cc:1145] Deleting tablet ff1b3ee229c445b19405cf4a19cf8003
0112 02:52:42.653023 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.653101 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1599228}
I20250112 02:52:42.653513 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 295) took 1532 ms. Trace:
W20250112 02:52:42.653525 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 3b9280db29e54d41af1bed2fa5aa5535 because the tablet was not found. No further retry: Not found: Tablet not found: 3b9280db29e54d41af1bed2fa5aa5535
I20250112 02:52:42.653573 2024 rpcz_store.cc:276] 0112 02:52:41.120982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.121041 (+ 59us) service_pool.cc:225] Handling call
0112 02:52:42.653416 (+1532375us) ts_tablet_manager.cc:1145] Deleting tablet d0cb54003a83472cb9c57ace3e65b0be
0112 02:52:42.653425 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.653497 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1532150}
I20250112 02:52:42.653820 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 296) took 1512 ms. Trace:
I20250112 02:52:42.653877 2024 rpcz_store.cc:276] 0112 02:52:41.141054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.141104 (+ 50us) service_pool.cc:225] Handling call
0112 02:52:42.653737 (+1512633us) ts_tablet_manager.cc:1145] Deleting tablet a8e7882e07784bdda18dc1ffd20963c8
0112 02:52:42.653744 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.653808 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1512441}
I20250112 02:52:42.654132 2024 tablet_replica.cc:331] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:42.654299 2024 raft_consensus.cc:2238] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250112 02:52:42.654294 855 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 46100.6us
I20250112 02:52:42.654431 2024 raft_consensus.cc:2267] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:42.654564 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet ff1b3ee229c445b19405cf4a19cf8003 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff1b3ee229c445b19405cf4a19cf8003 already in progress: creating tablet
W20250112 02:52:42.654681 855 leader_election.cc:343] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:42.654771 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet d0cb54003a83472cb9c57ace3e65b0be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0cb54003a83472cb9c57ace3e65b0be already in progress: creating tablet
I20250112 02:52:42.654784 855 leader_election.cc:304] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:42.654932 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet a8e7882e07784bdda18dc1ffd20963c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8e7882e07784bdda18dc1ffd20963c8 already in progress: creating tablet
I20250112 02:52:42.655159 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:42.655526 2876 raft_consensus.cc:2743] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:42.655761 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d3664071b146a3ac7c6885f9bda789" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:42.655992 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_5d473cabff2e46e0b531fbce28d9d558 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 1 got 0
I20250112 02:52:42.656200 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
W20250112 02:52:42.656335 855 leader_election.cc:343] T 54d3664071b146a3ac7c6885f9bda789 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.658813 2673 raft_consensus.cc:2798] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:42.658881 2673 raft_consensus.cc:491] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:42.658928 2673 raft_consensus.cc:3054] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.660043 2228 raft_consensus.cc:491] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.660146 2228 raft_consensus.cc:513] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.660641 2228 leader_election.cc:290] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.661057 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad9c47fcb534619949971796b60eba2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:42.661196 774 raft_consensus.cc:2463] T 7ad9c47fcb534619949971796b60eba2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:42.661175 2673 raft_consensus.cc:513] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.661536 991 leader_election.cc:304] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:42.661652 2673 leader_election.cc:290] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:42.661880 2228 raft_consensus.cc:2798] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:42.661938 2228 raft_consensus.cc:491] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:42.662009 2228 raft_consensus.cc:3054] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.663061 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37862023095644bab2403576e8cb8c58" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:42.663204 779 raft_consensus.cc:3054] T 37862023095644bab2403576e8cb8c58 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.664475 2228 raft_consensus.cc:513] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.664992 2228 leader_election.cc:290] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.665797 779 raft_consensus.cc:2463] T 37862023095644bab2403576e8cb8c58 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:42.666181 855 leader_election.cc:304] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:42.666580 2853 tablet.cc:1721] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:42.666683 2673 raft_consensus.cc:2798] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:42.666846 2673 raft_consensus.cc:695] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:42.667037 2673 consensus_queue.cc:237] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.667621 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad9c47fcb534619949971796b60eba2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:42.667784 779 raft_consensus.cc:3054] T 7ad9c47fcb534619949971796b60eba2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.670522 2853 tablet.cc:1721] T cf54f5185a0f4c70b9057309fca14e56 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:42.671113 779 raft_consensus.cc:2463] T 7ad9c47fcb534619949971796b60eba2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:42.671068 2865 tablet.cc:1721] T 777272e7159b438ebff4db93698411d0 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:42.671561 991 leader_election.cc:304] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:42.671816 2745 raft_consensus.cc:2798] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:42.671926 2745 raft_consensus.cc:695] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:42.672113 2745 consensus_queue.cc:237] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.674391 1181 tablet_bootstrap.cc:492] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:42.674490 1181 ts_tablet_manager.cc:1397] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.000s
I20250112 02:52:42.675071 1181 raft_consensus.cc:357] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.682924 1181 raft_consensus.cc:383] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.683163 1181 raft_consensus.cc:738] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.683676 1181 consensus_queue.cc:260] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.683784 1173 tablet_bootstrap.cc:492] T 41d5799b5b36487da42dc43b1c70ae26 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:42.683882 1173 ts_tablet_manager.cc:1397] T 41d5799b5b36487da42dc43b1c70ae26 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.112s user 0.003s sys 0.000s
I20250112 02:52:42.684418 1173 raft_consensus.cc:357] T 41d5799b5b36487da42dc43b1c70ae26 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.684587 1173 raft_consensus.cc:383] T 41d5799b5b36487da42dc43b1c70ae26 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.684650 1173 raft_consensus.cc:738] T 41d5799b5b36487da42dc43b1c70ae26 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
W20250112 02:52:42.677886 854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 77077us
I20250112 02:52:42.685173 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d3664071b146a3ac7c6885f9bda789" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:42.685336 2024 ts_tablet_manager.cc:1905] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:42.685523 2854 tablet.cc:1721] T cf54f5185a0f4c70b9057309fca14e56 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:42.687155 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37862023095644bab2403576e8cb8c58" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:42.687309 1051 raft_consensus.cc:3054] T 37862023095644bab2403576e8cb8c58 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.689877 1051 raft_consensus.cc:2463] T 37862023095644bab2403576e8cb8c58 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:42.691098 1173 consensus_queue.cc:260] T 41d5799b5b36487da42dc43b1c70ae26 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.692680 1173 ts_tablet_manager.cc:1428] T 41d5799b5b36487da42dc43b1c70ae26 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250112 02:52:42.692929 1173 tablet_bootstrap.cc:492] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:42.694398 1173 tablet_bootstrap.cc:654] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.699995 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad9c47fcb534619949971796b60eba2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:42.700160 914 raft_consensus.cc:2463] T 7ad9c47fcb534619949971796b60eba2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:42.700556 2787 raft_consensus.cc:491] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.700659 2787 raft_consensus.cc:513] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.700824 1181 ts_tablet_manager.cc:1428] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.026s user 0.001s sys 0.000s
I20250112 02:52:42.701049 1181 tablet_bootstrap.cc:492] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:42.701267 2787 leader_election.cc:290] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:42.702399 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7717cdcb8a4f47b11ba7af54b57e3c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:42.702839 855 leader_election.cc:343] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.704577 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad9c47fcb534619949971796b60eba2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:42.704734 916 raft_consensus.cc:3054] T 7ad9c47fcb534619949971796b60eba2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.707126 916 raft_consensus.cc:2463] T 7ad9c47fcb534619949971796b60eba2 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:42.716513 2854 tablet.cc:1721] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:42.716715 2024 ts_tablet_manager.cc:1918] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:42.716806 2024 log.cc:1198] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/bd5b3c712c0f448ea9b14ab93dd731ab
I20250112 02:52:42.718127 2855 tablet.cc:1721] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:42.718273 2853 tablet.cc:1721] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:42.718447 2865 tablet.cc:1721] T edfd1ebe729d458ea31eb7feff7ee072 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:42.719884 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3dab0466476c4a9aa265271c1abd91a4 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:42.720121 2024 ts_tablet_manager.cc:1939] T bd5b3c712c0f448ea9b14ab93dd731ab P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:42.720362 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 297) took 1578 ms. Trace:
I20250112 02:52:42.720456 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7717cdcb8a4f47b11ba7af54b57e3c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:42.720422 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_a9907ed8bfe14625977ec98b67e9d599 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 2 got 0
W20250112 02:52:42.720885 854 leader_election.cc:343] T 54d3664071b146a3ac7c6885f9bda789 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.720978 854 leader_election.cc:304] T 54d3664071b146a3ac7c6885f9bda789 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:42.721048 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_b9dff6cb7bbd4d0a88de458399b0d2c0 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 2 got 0
I20250112 02:52:42.721406 2787 raft_consensus.cc:2743] T 54d3664071b146a3ac7c6885f9bda789 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:42.721510 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ca2165362ee541adac724b63f39b1489 (table default.Table_41b7056b6e9240928d0af52aec2e9338 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 2 got 0
W20250112 02:52:42.722018 854 leader_election.cc:343] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.720429 2024 rpcz_store.cc:276] 0112 02:52:41.141957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.142023 (+ 66us) service_pool.cc:225] Handling call
0112 02:52:42.654096 (+1512073us) ts_tablet_manager.cc:1145] Deleting tablet bd5b3c712c0f448ea9b14ab93dd731ab
0112 02:52:42.654104 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.702149 (+ 48045us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.716640 (+ 14491us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.720347 (+ 3707us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1510798}
I20250112 02:52:42.722132 854 leader_election.cc:304] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:42.722410 2787 raft_consensus.cc:2743] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:42.722467 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet bd5b3c712c0f448ea9b14ab93dd731ab (table default.Table_9c10574f7c7d435890eb9a24af23bcb9 [id=c0e16b77525146a28b8182137bc81566]) successfully deleted
I20250112 02:52:42.723116 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 298) took 1578 ms. Trace:
I20250112 02:52:42.723191 2024 rpcz_store.cc:276] 0112 02:52:41.145050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.145098 (+ 48us) service_pool.cc:225] Handling call
0112 02:52:42.723002 (+1577904us) ts_tablet_manager.cc:1145] Deleting tablet c29a9ca43e034b43a00207e5847bbee3
0112 02:52:42.723011 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.723100 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1577228}
W20250112 02:52:42.723837 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet c29a9ca43e034b43a00207e5847bbee3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c29a9ca43e034b43a00207e5847bbee3 already in progress: creating tablet
I20250112 02:52:42.725733 1177 tablet_bootstrap.cc:492] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:42.725912 1177 ts_tablet_manager.cc:1397] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.082s user 0.002s sys 0.001s
I20250112 02:52:42.726686 1181 tablet_bootstrap.cc:654] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.727149 1177 raft_consensus.cc:357] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.727420 1177 raft_consensus.cc:383] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.727506 1177 raft_consensus.cc:738] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.727803 1177 consensus_queue.cc:260] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.727995 2451 raft_consensus.cc:491] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.728135 2451 raft_consensus.cc:513] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.728798 2451 leader_election.cc:290] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.729672 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f776e2a6ec704cf9b64483bc7d5b1dff" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:42.730073 714 leader_election.cc:343] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.730458 1177 ts_tablet_manager.cc:1428] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:42.733704 2855 tablet.cc:1721] T edfd1ebe729d458ea31eb7feff7ee072 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:42.737260 2908 tablet.cc:1721] T cf54f5185a0f4c70b9057309fca14e56 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:42.742492 2900 tablet.cc:1721] T edfd1ebe729d458ea31eb7feff7ee072 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:42.743691 2855 tablet.cc:1716] T 5d0c3191973a477e949fbf9dfcca963f P fe36951a83fb4e57b26e187edbcbfe3d: Skipping requested alter to schema version 3, tablet already version 3
I20250112 02:52:42.744486 2855 tablet.cc:1721] T 777272e7159b438ebff4db93698411d0 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:42.749773 2024 tablet_replica.cc:331] T b1d9fae290ec4281839991f9d40b105c P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:42.749970 2024 raft_consensus.cc:2238] T b1d9fae290ec4281839991f9d40b105c P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:42.750118 2024 raft_consensus.cc:2267] T b1d9fae290ec4281839991f9d40b105c P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:42.750597 2024 ts_tablet_manager.cc:1905] T b1d9fae290ec4281839991f9d40b105c P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:42.756934 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:42.759835 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:42.207726 2283 rpcz_store.cc:276] 0112 02:52:37.189095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:37.189145 (+ 50us) service_pool.cc:225] Handling call
0112 02:52:41.637148 (+4448003us) raft_consensus.cc:1466] Updating replica for 101 ops
0112 02:52:41.735093 (+ 97945us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.735101 (+ 8us) write_op.cc:454] Released schema lock
0112 02:52:41.737804 (+ 2703us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.737808 (+ 4us) write_op.cc:454] Released schema lock
0112 02:52:41.738315 (+ 507us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.738318 (+ 3us) write_op.cc:454] Released schema lock
0112 02:52:41.742934 (+ 4616us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.742937 (+ 3us) write_op.cc:454] Released schema lock
0112 02:52:41.743742 (+ 805us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.743745 (+ 3us) write_op.cc:454] Released schema lock
0112 02:52:41.743984 (+ 239us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.743987 (+ 3us) write_op.cc:454] Released schema lock
0112 02:52:41.744498 (+ 511us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.744549 (+ 51us) write_op.cc:454] Released schema lock
0112 02:52:41.745721 (+ 1172us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.745726 (+ 5us) write_op.cc:454] Released schema lock
0112 02:52:41.747588 (+ 1862us) raft_consensus.cc:1529] Early marking committed up to index 514
0112 02:52:41.751721 (+ 4133us) raft_consensus.cc:1534] Triggering prepare for 101 ops
0112 02:52:41.752086 (+ 365us) write_op.cc:270] Start()
0112 02:52:41.752163 (+ 77us) write_op.cc:276] Timestamp: P: 1736650352984830 usec, L: 0
0112 02:52:41.766125 (+ 13962us) write_op.cc:270] Start()
0112 02:52:41.766160 (+ 35us) write_op.cc:276] Timestamp: P: 1736650353083314 usec, L: 0
0112 02:52:41.766334 (+ 174us) write_op.cc:270] Start()
0112 02:52:41.766352 (+ 18us) write_op.cc:276] Timestamp: P: 1736650353083970 usec, L: 0
0112 02:52:41.766509 (+ 157us) write_op.cc:270] Start()
0112 02:52:41.766527 (+ 18us) write_op.cc:276] Timestamp: P: 1736650353167371 usec, L: 0
0112 02:52:41.766680 (+ 153us) write_op.cc:270] Start()
0112 02:52:41.766699 (+ 19us) write_op.cc:276] Timestamp: P: 1736650353168484 usec, L: 0
0112 02:52:41.767055 (+ 356us) write_op.cc:270] Start()
0112 02:52:41.767077 (+ 22us) write_op.cc:276] Timestamp: P: 1736650353169238 usec, L: 0
0112 02:52:41.767251 (+ 174us) write_op.cc:270] Start()
0112 02:52:41.767825 (+ 574us) write_op.cc:276] Timestamp: P: 1736650353170040 usec, L: 0
0112 02:52:41.768050 (+ 225us) write_op.cc:270] Start()
0112 02:52:41.768076 (+ 26us) write_op.cc:276] Timestamp: P: 1736650353171108 usec, L: 0
0112 02:52:41.768237 (+ 161us) write_op.cc:270] Start()
0112 02:52:41.768256 (+ 19us) write_op.cc:276] Timestamp: P: 1736650353175025 usec, L: 0
0112 02:52:41.768460 (+ 204us) write_op.cc:270] Start()
0112 02:52:41.768478 (+ 18us) write_op.cc:276] Timestamp: P: 1736650353181718 usec, L: 0
0112 02:52:41.768684 (+ 206us) write_op.cc:270] Start()
0112 02:52:41.768703 (+ 19us) write_op.cc:276] Timestamp: P: 1736650353182389 usec, L: 0
0112 02:52:41.768847 (+ 144us) write_op.cc:270] Start()
0112 02:52:41.768867 (+ 20us) write_op.cc:276] Timestamp: P: 1736650353263403 usec, L: 0
0112 02:52:41.768994 (+ 127us) write_op.cc:270] Start()
0112 02:52:41.769013 (+ 19us) write_op.cc:276] Timestamp: P: 1736650353268066 usec, L: 0
0112 02:52:41.769151 (+ 138us) write_op.cc:270] Start()
0112 02:52:41.769170 (+ 19us) write_op.cc:276] Timestamp: P: 1736650353273348 usec, L: 0
0112 02:52:41.769438 (+ 268us) write_op.cc:270] Start()
0112 02:52:41.769463 (+ 25us) write_op.cc:276] Timestamp: P: 1736650353279771 usec, L: 0
0112 02:52:41.769627 (+ 164us) write_op.cc:270] Start()
0112 02:52:41.769653 (+ 26us) write_op.cc:276] Timestamp: P: 1736650353280234 usec, L: 0
0112 02:52:41.769786 (+ 133us) write_op.cc:270] Start()
0112 02:52:41.769803 (+ 17us) write_op.cc:276] Timestamp: P: 1736650353303192 usec, L: 0
0112 02:52:41.769924 (+ 121us) write_op.cc:270] Start()
0112 02:52:41.769943 (+ 19us) write_op.cc:276] Timestamp: P: 1736650353310492 usec, L: 0
0112 02:52:41.770086 (+ 143us) write_op.cc:270] Start()
0112 02:52:41.770102 (+ 16us) write_op.cc:276] Timestamp: P: 1736650353321253 usec, L: 0
0112 02:52:41.770254 (+ 152us) write_op.cc:270] Start()
0112 02:52:41.770272 (+ 18us) write_op.cc:276] Timestamp: P: 1736650353709719 usec, L: 0
0112 02:52:41.770390 (+ 118us) write_op.cc:270] Start()
0112 02:52:41.770412 (+ 22us) write_op.cc:276] Timestamp: P: 1736650353777799 usec, L: 0
0112 02:52:41.770583 (+ 171us) write_op.cc:270] Start()
0112 02:52:41.770601 (+ 18us) write_op.cc:276] Timestamp: P: 1736650353778481 usec, L: 0
0112 02:52:41.770726 (+ 125us) write_op.cc:270] Start()
0112 02:52:41.770744 (+ 18us) write_op.cc:276] Timestamp: P: 1736650353868800 usec, L: 0
0112 02:52:41.770853 (+ 109us) write_op.cc:270] Start()
0112 02:52:41.770869 (+ 16us) write_op.cc:276] Timestamp: P: 1736650353871810 usec, L: 0
0112 02:52:41.771039 (+ 170us) write_op.cc:270] Start()
0112 02:52:41.771057 (+ 18us) write_op.cc:276] Timestamp: P: 1736650353873981 usec, L: 0
0112 02:52:41.771192 (+ 135us) write_op.cc:270] Start()
0112 02:52:41.771208 (+ 16us) write_op.cc:276] Timestamp: P: 1736650353878577 usec, L: 0
0112 02:52:41.771340 (+ 132us) write_op.cc:270] Start()
0112 02:52:41.771356 (+ 16us) write_op.cc:276] Timestamp: P: 1736650353879093 usec, L: 0
0112 02:52:41.771493 (+ 137us) write_op.cc:270] Start()
0112 02:52:41.771512 (+ 19us) write_op.cc:276] Timestamp: P: 1736650353885796 usec, L: 0
0112 02:52:41.804740 (+ 33228us) write_op.cc:270] Start()
0112 02:52:41.804766 (+ 26us) write_op.cc:276] Timestamp: P: 1736650353886397 usec, L: 0
0112 02:52:41.804924 (+ 158us) write_op.cc:270] Start()
0112 02:52:41.804943 (+ 19us) write_op.cc:276] Timestamp: P: 1736650353893622 usec, L: 0
0112 02:52:41.805068 (+ 125us) write_op.cc:270] Start()
0112 02:52:41.805086 (+ 18us) write_op.cc:276] Timestamp: P: 1736650353902246 usec, L: 0
0112 02:52:41.805232 (+ 146us) write_op.cc:270] Start()
0112 02:52:41.805251 (+ 19us) write_op.cc:276] Timestamp: P: 1736650353916449 usec, L: 0
0112 02:52:41.805429 (+ 178us) write_op.cc:270] Start()
0112 02:52:41.805448 (+ 19us) write_op.cc:276] Timestamp: P: 1736650353917582 usec, L: 0
0112 02:52:41.805576 (+ 128us) write_op.cc:270] Start()
0112 02:52:41.805595 (+ 19us) write_op.cc:276] Timestamp: P: 1736650353918550 usec, L: 0
0112 02:52:41.861776 (+ 56181us) write_op.cc:270] Start()
0112 02:52:41.863377 (+ 1601us) write_op.cc:276] Timestamp: P: 1736650353919553 usec, L: 0
0112 02:52:41.907165 (+ 43788us) write_op.cc:270] Start()
0112 02:52:41.907390 (+ 225us) write_op.cc:276] Timestamp: P: 1736650353935638 usec, L: 0
0112 02:52:41.909491 (+ 2101us) write_op.cc:270] Start()
0112 02:52:41.909719 (+ 228us) write_op.cc:276] Timestamp: P: 1736650353945678 usec, L: 0
0112 02:52:41.911616 (+ 1897us) write_op.cc:270] Start()
0112 02:52:41.911789 (+ 173us) write_op.cc:276] Timestamp: P: 1736650353947009 usec, L: 0
0112 02:52:41.981600 (+ 69811us) write_op.cc:270] Start()
0112 02:52:41.981636 (+ 36us) write_op.cc:276] Timestamp: P: 1736650354004463 usec, L: 0
0112 02:52:41.981822 (+ 186us) write_op.cc:270] Start()
0112 02:52:41.981867 (+ 45us) write_op.cc:276] Timestamp: P: 1736650354023591 usec, L: 0
0112 02:52:41.982034 (+ 167us) write_op.cc:270] Start()
0112 02:52:41.982053 (+ 19us) write_op.cc:276] Timestamp: P: 1736650354025149 usec, L: 0
0112 02:52:41.982285 (+ 232us) write_op.cc:270] Start()
0112 02:52:41.982307 (+ 22us) write_op.cc:276] Timestamp: P: 1736650354057675 usec, L: 0
0112 02:52:41.982489 (+ 182us) write_op.cc:270] Start()
0112 02:52:41.982511 (+ 22us) write_op.cc:276] Timestamp: P: 1736650354069134 usec, L: 0
0112 02:52:41.982654 (+ 143us) write_op.cc:270] Start()
0112 02:52:41.982677 (+ 23us) write_op.cc:276] Timestamp: P: 1736650354070157 usec, L: 0
0112 02:52:41.982898 (+ 221us) write_op.cc:270] Start()
0112 02:52:41.982933 (+ 35us) write_op.cc:276] Timestamp: P: 1736650354072735 usec, L: 0
0112 02:52:41.983103 (+ 170us) write_op.cc:270] Start()
0112 02:52:41.983129 (+ 26us) write_op.cc:276] Timestamp: P: 1736650354073268 usec, L: 0
0112 02:52:41.983272 (+ 143us) write_op.cc:270] Start()
0112 02:52:41.983304 (+ 32us) write_op.cc:276] Timestamp: P: 1736650354077397 usec, L: 0
0112 02:52:41.983491 (+ 187us) write_op.cc:270] Start()
0112 02:52:41.983511 (+ 20us) write_op.cc:276] Timestamp: P: 1736650354084732 usec, L: 0
0112 02:52:41.984365 (+ 854us) write_op.cc:270] Start()
0112 02:52:41.984389 (+ 24us) write_op.cc:276] Timestamp: P: 1736650354229949 usec, L: 0
0112 02:52:41.984611 (+ 222us) write_op.cc:270] Start()
0112 02:52:41.984632 (+ 21us) write_op.cc:276] Timestamp: P: 1736650354231223 usec, L: 0
0112 02:52:41.984805 (+ 173us) write_op.cc:270] Start()
0112 02:52:41.984843 (+ 38us) write_op.cc:276] Timestamp: P: 1736650354335798 usec, L: 0
0112 02:52:42.003178 (+ 18335us) write_op.cc:270] Start()
0112 02:52:42.003228 (+ 50us) write_op.cc:276] Timestamp: P: 1736650354337472 usec, L: 0
0112 02:52:42.003442 (+ 214us) write_op.cc:270] Start()
0112 02:52:42.003461 (+ 19us) write_op.cc:276] Timestamp: P: 1736650354407416 usec, L: 0
0112 02:52:42.003591 (+ 130us) write_op.cc:270] Start()
0112 02:52:42.003607 (+ 16us) write_op.cc:276] Timestamp: P: 1736650354408946 usec, L: 0
0112 02:52:42.003711 (+ 104us) write_op.cc:270] Start()
0112 02:52:42.003727 (+ 16us) write_op.cc:276] Timestamp: P: 1736650354415870 usec, L: 0
0112 02:52:42.003824 (+ 97us) write_op.cc:270] Start()
0112 02:52:42.003839 (+ 15us) write_op.cc:276] Timestamp: P: 1736650354417999 usec, L: 0
0112 02:52:42.003948 (+ 109us) write_op.cc:270] Start()
0112 02:52:42.003964 (+ 16us) write_op.cc:276] Timestamp: P: 1736650354419186 usec, L: 0
0112 02:52:42.004071 (+ 107us) write_op.cc:270] Start()
0112 02:52:42.004116 (+ 45us) write_op.cc:276] Timestamp: P: 1736650354427104 usec, L: 0
0112 02:52:42.004225 (+ 109us) write_op.cc:270] Start()
0112 02:52:42.004241 (+ 16us) write_op.cc:276] Timestamp: P: 1736650354435219 usec, L: 0
0112 02:52:42.004343 (+ 102us) write_op.cc:270] Start()
0112 02:52:42.004361 (+ 18us) write_op.cc:276] Timestamp: P: 1736650354437933 usec, L: 0
0112 02:52:42.004477 (+ 116us) write_op.cc:270] Start()
0112 02:52:42.004498 (+ 21us) write_op.cc:276] Timestamp: P: 1736650354462627 usec, L: 0
0112 02:52:42.004639 (+ 141us) write_op.cc:270] Start()
0112 02:52:42.004658 (+ 19us) write_op.cc:276] Timestamp: P: 1736650354468794 usec, L: 0
0112 02:52:42.004787 (+ 129us) write_op.cc:270] Start()
0112 02:52:42.004807 (+ 20us) write_op.cc:276] Timestamp: P: 1736650354499805 usec, L: 0
0112 02:52:42.004949 (+ 142us) write_op.cc:270] Start()
0112 02:52:42.004970 (+ 21us) write_op.cc:276] Timestamp: P: 1736650354547373 usec, L: 0
0112 02:52:42.005370 (+ 400us) write_op.cc:270] Start()
0112 02:52:42.005397 (+ 27us) write_op.cc:276] Timestamp: P: 1736650354550092 usec, L: 0
0112 02:52:42.005560 (+ 163us) write_op.cc:270] Start()
0112 02:52:42.005582 (+ 22us) write_op.cc:276] Timestamp: P: 1736650354552347 usec, L: 0
0112 02:52:42.005709 (+ 127us) write_op.cc:270] Start()
0112 02:52:42.005729 (+ 20us) write_op.cc:276] Timestamp: P: 1736650354559791 usec, L: 0
0112 02:52:42.005904 (+ 175us) write_op.cc:270] Start()
0112 02:52:42.005957 (+ 53us) write_op.cc:276] Timestamp: P: 1736650354565369 usec, L: 0
0112 02:52:42.038838 (+ 32881us) write_op.cc:270] Start()
0112 02:52:42.038875 (+ 37us) write_op.cc:276] Timestamp: P: 1736650354569161 usec, L: 0
0112 02:52:42.039093 (+ 218us) write_op.cc:270] Start()
0112 02:52:42.039122 (+ 29us) write_op.cc:276] Timestamp: P: 1736650354603943 usec, L: 0
0112 02:52:42.039310 (+ 188us) write_op.cc:270] Start()
0112 02:52:42.039333 (+ 23us) write_op.cc:276] Timestamp: P: 1736650354629680 usec, L: 0
0112 02:52:42.039456 (+ 123us) write_op.cc:270] Start()
0112 02:52:42.039515 (+ 59us) write_op.cc:276] Timestamp: P: 1736650354638649 usec, L: 0
0112 02:52:42.039669 (+ 154us) write_op.cc:270] Start()
0112 02:52:42.039689 (+ 20us) write_op.cc:276] Timestamp: P: 1736650354663039 usec, L: 0
0112 02:52:42.039814 (+ 125us) write_op.cc:270] Start()
0112 02:52:42.039836 (+ 22us) write_op.cc:276] Timestamp: P: 1736650354757242 usec, L: 0
0112 02:52:42.039964 (+ 128us) write_op.cc:270] Start()
0112 02:52:42.039985 (+ 21us) write_op.cc:276] Timestamp: P: 1736650354776693 usec, L: 0
0112 02:52:42.040154 (+ 169us) write_op.cc:270] Start()
0112 02:52:42.040183 (+ 29us) write_op.cc:276] Timestamp: P: 1736650354788254 usec, L: 0
0112 02:52:42.040317 (+ 134us) write_op.cc:270] Start()
0112 02:52:42.040335 (+ 18us) write_op.cc:276] Timestamp: P: 1736650354819326 usec, L: 0
0112 02:52:42.040453 (+ 118us) write_op.cc:270] Start()
0112 02:52:42.040472 (+ 19us) write_op.cc:276] Timestamp: P: 1736650354849291 usec, L: 0
0112 02:52:42.040603 (+ 131us) write_op.cc:270] Start()
0112 02:52:42.040656 (+ 53us) write_op.cc:276] Timestamp: P: 1736650354898213 usec, L: 0
0112 02:52:42.040796 (+ 140us) write_op.cc:270] Start()
0112 02:52:42.040816 (+ 20us) write_op.cc:276] Timestamp: P: 1736650354944680 usec, L: 0
0112 02:52:42.040936 (+ 120us) write_op.cc:270] Start()
0112 02:52:42.040956 (+ 20us) write_op.cc:276] Timestamp: P: 1736650354987397 usec, L: 0
0112 02:52:42.041096 (+ 140us) write_op.cc:270] Start()
0112 02:52:42.041115 (+ 19us) write_op.cc:276] Timestamp: P: 1736650355034063 usec, L: 0
0112 02:52:42.041358 (+ 243us) write_op.cc:270] Start()
0112 02:52:42.041392 (+ 34us) write_op.cc:276] Timestamp: P: 1736650355093933 usec, L: 0
0112 02:52:42.041563 (+ 171us) write_op.cc:270] Start()
0112 02:52:42.041583 (+ 20us) write_op.cc:276] Timestamp: P: 1736650355858676 usec, L: 0
0112 02:52:42.041714 (+ 131us) write_op.cc:270] Start()
0112 02:52:42.041731 (+ 17us) write_op.cc:276] Timestamp: P: 1736650356215143 usec, L: 0
0112 02:52:42.041837 (+ 106us) write_op.cc:270] Start()
0112 02:52:42.041854 (+ 17us) write_op.cc:276] Timestamp: P: 1736650356228862 usec, L: 0
0112 02:52:42.042027 (+ 173us) write_op.cc:270] Start()
0112 02:52:42.042050 (+ 23us) write_op.cc:276] Timestamp: P: 1736650356306336 usec, L: 0
0112 02:52:42.042205 (+ 155us) write_op.cc:270] Start()
0112 02:52:42.042225 (+ 20us) write_op.cc:276] Timestamp: P: 1736650356328197 usec, L: 0
0112 02:52:42.042361 (+ 136us) write_op.cc:270] Start()
0112 02:52:42.042379 (+ 18us) write_op.cc:276] Timestamp: P: 1736650356387229 usec, L: 0
0112 02:52:42.042515 (+ 136us) write_op.cc:270] Start()
0112 02:52:42.042534 (+ 19us) write_op.cc:276] Timestamp: P: 1736650356427866 usec, L: 0
0112 02:52:42.067887 (+ 25353us) write_op.cc:270] Start()
0112 02:52:42.067944 (+ 57us) write_op.cc:276] Timestamp: P: 1736650356441741 usec, L: 0
0112 02:52:42.068171 (+ 227us) write_op.cc:270] Start()
0112 02:52:42.068193 (+ 22us) write_op.cc:276] Timestamp: P: 1736650356462035 usec, L: 0
0112 02:52:42.068323 (+ 130us) write_op.cc:270] Start()
0112 02:52:42.068341 (+ 18us) write_op.cc:276] Timestamp: P: 1736650356669550 usec, L: 0
0112 02:52:42.068466 (+ 125us) write_op.cc:270] Start()
0112 02:52:42.068484 (+ 18us) write_op.cc:276] Timestamp: P: 1736650356684668 usec, L: 0
0112 02:52:42.068615 (+ 131us) write_op.cc:270] Start()
0112 02:52:42.068634 (+ 19us) write_op.cc:276] Timestamp: P: 1736650356694115 usec, L: 0
0112 02:52:42.068746 (+ 112us) write_op.cc:270] Start()
0112 02:52:42.068765 (+ 19us) write_op.cc:276] Timestamp: P: 1736650356704338 usec, L: 0
0112 02:52:42.068888 (+ 123us) write_op.cc:270] Start()
0112 02:52:42.068905 (+ 17us) write_op.cc:276] Timestamp: P: 1736650356732422 usec, L: 0
0112 02:52:42.069048 (+ 143us) write_op.cc:270] Start()
0112 02:52:42.069068 (+ 20us) write_op.cc:276] Timestamp: P: 1736650356765299 usec, L: 0
0112 02:52:42.069229 (+ 161us) write_op.cc:270] Start()
0112 02:52:42.069251 (+ 22us) write_op.cc:276] Timestamp: P: 1736650356801951 usec, L: 0
0112 02:52:42.069689 (+ 438us) write_op.cc:270] Start()
0112 02:52:42.069715 (+ 26us) write_op.cc:276] Timestamp: P: 1736650356892483 usec, L: 0
0112 02:52:42.077177 (+ 7462us) log.cc:844] Serialized 75304 byte log entry
0112 02:52:42.083119 (+ 5942us) raft_consensus.cc:1640] Marking committed up to 615
0112 02:52:42.083222 (+ 103us) log.cc:844] Serialized 15 byte log entry
0112 02:52:42.112513 (+ 29291us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.112518 (+ 5us) write_op.cc:454] Released schema lock
0112 02:52:42.113303 (+ 785us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.113308 (+ 5us) write_op.cc:454] Released schema lock
0112 02:52:42.114005 (+ 697us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.114010 (+ 5us) write_op.cc:454] Released schema lock
0112 02:52:42.114889 (+ 879us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.114896 (+ 7us) write_op.cc:454] Released schema lock
0112 02:52:42.118072 (+ 3176us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.118077 (+ 5us) write_op.cc:454] Released schema lock
0112 02:52:42.118803 (+ 726us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.118808 (+ 5us) write_op.cc:454] Released schema lock
0112 02:52:42.119304 (+ 496us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.119308 (+ 4us) write_op.cc:454] Released schema lock
0112 02:52:42.119959 (+ 651us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.119964 (+ 5us) write_op.cc:454] Released schema lock
0112 02:52:42.137095 (+ 17131us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.137103 (+ 8us) write_op.cc:454] Released schema lock
0112 02:52:42.146451 (+ 9348us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.146456 (+ 5us) write_op.cc:454] Released schema lock
0112 02:52:42.150264 (+ 3808us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.163243 (+ 12979us) write_op.cc:454] Released schema lock
0112 02:52:42.164380 (+ 1137us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.164385 (+ 5us) write_op.cc:454] Released schema lock
0112 02:52:42.165290 (+ 905us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.165295 (+ 5us) write_op.cc:454] Released schema lock
0112 02:52:42.182437 (+ 17142us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.182444 (+ 7us) write_op.cc:454] Released schema lock
0112 02:52:42.185033 (+ 2589us) raft_consensus.cc:1692] Filling consensus response to leader.
0112 02:52:42.185648 (+ 615us) spinlock_profiling.cc:243] Waited 439 ms on lock 0x616000050854. stack: 00007f1550951fd6 00007f1550951d90 00007f154f93148d 000000000096d544 000000000096d50d 00007f155f4efe1d 00007f155f4eff13 00007f155c1f8211 00007f1563eb6645 00007f1563ec97a5 00007f1563ec9551 00007f155d519d0d 00007f155c208e9e 00007f155c208bed 0000000000a71ecc 00007f1550998b64
0112 02:52:42.191710 (+ 6062us) spinlock_profiling.cc:243] Waited 96.3 ms on lock 0x616000050854. stack: 00007f1550951fd6 00007f1550951d90 00007f154f93148d 000000000096d544 000000000096d50d 00007f155f4efe1d 00007f155f4eff13 00007f155c1f8211 00007f1563eb6645 00007f1563ec97a5 00007f1563ec9551 00007f155d519d0d 00007f155c208e9e 00007f155c208bed 0000000000a71ecc 00007f1550998b64
0112 02:52:42.200399 (+ 8689us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0112 02:52:42.200425 (+ 26us) raft_consensus.cc:1680] finished
0112 02:52:42.200429 (+ 4us) raft_consensus.cc:1686] UpdateReplicas() finished
0112 02:52:42.200562 (+ 133us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:41.766008 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.766102 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.766107 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:41.766109 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:41.766189 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.766200 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.766204 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:41.766252 (+ 48us) tablet.cc:698] Row locks acquired
0112 02:52:41.766254 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:42.111770 (+345516us) op_driver.cc:464] REPLICATION: finished
0112 02:52:42.111869 (+ 99us) write_op.cc:301] APPLY: starting
0112 02:52:42.111885 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:42.111926 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:42.111929 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:42.111980 (+ 51us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:42.112012 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:42.112049 (+ 37us) write_op.cc:312] APPLY: finished
0112 02:52:42.112133 (+ 84us) log.cc:844] Serialized 23 byte log entry
0112 02:52:42.112238 (+ 105us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.112257 (+ 19us) write_op.cc:454] Released schema lock
0112 02:52:42.112264 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 02:52:41.766315 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.766389 (+ 74us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.766393 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:41.766395 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:41.766472 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.766486 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.766490 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:41.766539 (+ 49us) tablet.cc:698] Row locks acquired
0112 02:52:41.766541 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:42.112600 (+346059us) op_driver.cc:464] REPLICATION: finished
0112 02:52:42.112667 (+ 67us) write_op.cc:301] APPLY: starting
0112 02:52:42.112677 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:42.112709 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:42.112712 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:42.112813 (+ 101us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:42.112870 (+ 57us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:42.112903 (+ 33us) write_op.cc:312] APPLY: finished
0112 02:52:42.112965 (+ 62us) log.cc:844] Serialized 29 byte log entry
0112 02:52:42.113037 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.113053 (+ 16us) write_op.cc:454] Released schema lock
0112 02:52:42.113058 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 02:52:41.766617 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.766702 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.766706 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:41.766707 (+ 1us) tablet.cc:659] Decoding operations
0112 02:52:41.766776 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.766786 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.766790 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:41.766841 (+ 51us) tablet.cc:698] Row locks acquired
0112 02:52:41.766843 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:42.113387 (+346544us) op_driver.cc:464] REPLICATION: finished
0112 02:52:42.113455 (+ 68us) write_op.cc:301] APPLY: starting
0112 02:52:42.113465 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:42.113503 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:42.113506 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:42.113573 (+ 67us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:42.113638 (+ 65us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:42.113673 (+ 35us) write_op.cc:312] APPLY: finished
0112 02:52:42.113726 (+ 53us) log.cc:844] Serialized 29 byte log entry
0112 02:52:42.113781 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.113794 (+ 13us) write_op.cc:454] Released schema lock
0112 02:52:42.113800 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 02:52:41.766901 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.766970 (+ 69us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.766973 (+ 3us) write_op.cc:435] Acquired schema lock
0112 02:52:41.766975 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:41.767063 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.767072 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.767076 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:41.767131 (+ 55us) tablet.cc:698] Row locks acquired
0112 02:52:41.767133 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:42.114271 (+347138us) op_driver.cc:464] REPLICATION: finished
0112 02:52:42.114372 (+ 101us) write_op.cc:301] APPLY: starting
0112 02:52:42.114382 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:42.114406 (+ 24us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:42.114408 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:42.114442 (+ 34us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:42.114463 (+ 21us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:42.114487 (+ 24us) write_op.cc:312] APPLY: finished
0112 02:52:42.114533 (+ 46us) log.cc:844] Serialized 23 byte log entry
0112 02:52:42.114581 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.114592 (+ 11us) write_op.cc:454] Released schema lock
0112 02:52:42.114597 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 02:52:41.767181 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.767254 (+ 73us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.767258 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:41.767260 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:41.767326 (+ 66us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.767335 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.767339 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:41.767378 (+ 39us) tablet.cc:698] Row locks acquired
0112 02:52:41.767382 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:42.114961 (+347579us) op_driver.cc:464] REPLICATION: finished
0112 02:52:42.115032 (+ 71us) write_op.cc:301] APPLY: starting
0112 02:52:42.115042 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:42.115067 (+ 25us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:42.115069 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:42.115100 (+ 31us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:42.115121 (+ 21us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:42.115156 (+ 35us) write_op.cc:312] APPLY: finished
0112 02:52:42.115220 (+ 64us) log.cc:844] Serialized 23 byte log entry
0112 02:52:42.115273 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.115285 (+ 12us) write_op.cc:454] Released schema lock
0112 02:52:42.115290 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 02:52:41.767438 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.767531 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.767536 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:41.767537 (+ 1us) tablet.cc:659] Decoding operations
0112 02:52:41.767602 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.767611 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.767616 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:41.767674 (+ 58us) tablet.cc:698] Row locks acquired
0112 02:52:41.767676 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:42.118237 (+350561us) op_driver.cc:464] REPLICATION: finished
0112 02:52:42.118306 (+ 69us) write_op.cc:301] APPLY: starting
0112 02:52:42.11831
I20250112 02:52:42.752200 1173 tablet_bootstrap.cc:492] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:42.759600 2853 tablet.cc:1716] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6: Skipping requested alter to schema version 3, tablet already version 3
I20250112 02:52:42.769122 1173 ts_tablet_manager.cc:1397] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.000s
I20250112 02:52:42.769938 1173 raft_consensus.cc:357] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.770134 1173 raft_consensus.cc:383] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.770190 1173 raft_consensus.cc:738] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.771508 1173 consensus_queue.cc:260] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.772102 1173 ts_tablet_manager.cc:1428] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 02:52:42.774684 1181 tablet_bootstrap.cc:492] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:42.774786 1181 ts_tablet_manager.cc:1397] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.005s
I20250112 02:52:42.775326 1181 raft_consensus.cc:357] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.776085 2024 ts_tablet_manager.cc:1918] T b1d9fae290ec4281839991f9d40b105c P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:42.776185 2024 log.cc:1198] T b1d9fae290ec4281839991f9d40b105c P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/b1d9fae290ec4281839991f9d40b105c
I20250112 02:52:42.776434 2024 ts_tablet_manager.cc:1939] T b1d9fae290ec4281839991f9d40b105c P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:42.776991 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 299) took 1631 ms. Trace:
I20250112 02:52:42.777065 2024 rpcz_store.cc:276] 0112 02:52:41.145577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.145622 (+ 45us) service_pool.cc:225] Handling call
0112 02:52:42.723430 (+1577808us) ts_tablet_manager.cc:1145] Deleting tablet b1d9fae290ec4281839991f9d40b105c
0112 02:52:42.723452 (+ 22us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.773632 (+ 50180us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.776013 (+ 2381us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.776977 (+ 964us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1577625}
W20250112 02:52:42.732626 713 leader_election.cc:343] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: f776e2a6ec704cf9b64483bc7d5b1dff
I20250112 02:52:42.781237 713 leader_election.cc:304] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
W20250112 02:52:42.781514 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 48890.1us
I20250112 02:52:42.781963 1181 raft_consensus.cc:383] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.781972 2445 raft_consensus.cc:2743] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:42.782053 1181 raft_consensus.cc:738] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.782274 1181 consensus_queue.cc:260] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.734884 2865 tablet.cc:1716] T 5d0c3191973a477e949fbf9dfcca963f P 289afc116aa74408aa5bff74e11c50d8: Skipping requested alter to schema version 3, tablet already version 3
I20250112 02:52:42.785142 1181 ts_tablet_manager.cc:1428] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250112 02:52:42.786486 1181 tablet_bootstrap.cc:492] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:42.790050 1177 tablet_bootstrap.cc:492] T cec5346e13164274ae7bc3b6aa241f61 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:42.790657 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 300) took 1644 ms. Trace:
I20250112 02:52:42.790732 2024 rpcz_store.cc:276] 0112 02:52:41.146411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.146450 (+ 39us) service_pool.cc:225] Handling call
0112 02:52:42.790570 (+1644120us) ts_tablet_manager.cc:1145] Deleting tablet bab4978edea0445fb0456e88e7fb4c44
0112 02:52:42.790578 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.790641 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1642544}
I20250112 02:52:42.791121 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 301) took 1643 ms. Trace:
I20250112 02:52:42.791198 2024 rpcz_store.cc:276] 0112 02:52:41.147785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.147873 (+ 88us) service_pool.cc:225] Handling call
0112 02:52:42.791050 (+1643177us) ts_tablet_manager.cc:1145] Deleting tablet 8c821ea3ca4946c290c17135366e0cac
0112 02:52:42.791059 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.791108 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1642861}
I20250112 02:52:42.769083 2704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "436220c23fa948efb5b6ca15875d8b0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:42.777432 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet b1d9fae290ec4281839991f9d40b105c (table default.Table_c1639450ddf94784a40b8cf28f3059c4 [id=93cb911ed4e045c994c9a06b10e2e4a8]) successfully deleted
I20250112 02:52:42.791611 2704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [sys.catalog]: This master's current role is: LEADER
W20250112 02:52:42.791739 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet bab4978edea0445fb0456e88e7fb4c44 because the tablet was not found. No further retry: Not found: Tablet not found: bab4978edea0445fb0456e88e7fb4c44
W20250112 02:52:42.792002 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 8c821ea3ca4946c290c17135366e0cac because the tablet was not found. No further retry: Not found: Tablet not found: 8c821ea3ca4946c290c17135366e0cac
I20250112 02:52:42.792248 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 302) took 1642 ms. Trace:
I20250112 02:52:42.792366 2024 rpcz_store.cc:276] 0112 02:52:41.149450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.149508 (+ 58us) service_pool.cc:225] Handling call
0112 02:52:42.791505 (+1641997us) ts_tablet_manager.cc:1145] Deleting tablet f776e2a6ec704cf9b64483bc7d5b1dff
0112 02:52:42.791513 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.792230 (+ 717us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1641873}
I20250112 02:52:42.792631 1181 tablet_bootstrap.cc:654] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:42.793466 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f776e2a6ec704cf9b64483bc7d5b1dff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f776e2a6ec704cf9b64483bc7d5b1dff already in progress: creating tablet
I20250112 02:52:42.793872 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 305) took 1639 ms. Trace:
I20250112 02:52:42.793982 2024 rpcz_store.cc:276] 0112 02:52:41.154870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.154914 (+ 44us) service_pool.cc:225] Handling call
0112 02:52:42.793765 (+1638851us) ts_tablet_manager.cc:1145] Deleting tablet 62f4bd2df0ce45dabc55373f1154d3ac
0112 02:52:42.793776 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.793856 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1638704}
W20250112 02:52:42.795034 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 62f4bd2df0ce45dabc55373f1154d3ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62f4bd2df0ce45dabc55373f1154d3ac already in progress: creating tablet
I20250112 02:52:42.795418 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 306) took 1640 ms. Trace:
I20250112 02:52:42.795531 2024 rpcz_store.cc:276] 0112 02:52:41.155216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.155261 (+ 45us) service_pool.cc:225] Handling call
0112 02:52:42.795313 (+1640052us) ts_tablet_manager.cc:1145] Deleting tablet 82b351c563d749df806ab5d0331a4ef2
0112 02:52:42.795324 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.795403 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1639928}
W20250112 02:52:42.795938 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 82b351c563d749df806ab5d0331a4ef2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82b351c563d749df806ab5d0331a4ef2 already in progress: creating tablet
I20250112 02:52:42.799557 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 304) took 1644 ms. Trace:
I20250112 02:52:42.799665 2024 rpcz_store.cc:276] 0112 02:52:41.154755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.155414 (+ 659us) service_pool.cc:225] Handling call
0112 02:52:42.799440 (+1644026us) ts_tablet_manager.cc:1145] Deleting tablet f234256b343f447dadfc3d1936a824ee
0112 02:52:42.799450 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.799542 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1643854}
W20250112 02:52:42.800087 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f234256b343f447dadfc3d1936a824ee because the tablet was not found. No further retry: Not found: Tablet not found: f234256b343f447dadfc3d1936a824ee
I20250112 02:52:42.800393 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 303) took 1645 ms. Trace:
I20250112 02:52:42.800491 2024 rpcz_store.cc:276] 0112 02:52:41.154654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.155930 (+ 1276us) service_pool.cc:225] Handling call
0112 02:52:42.800320 (+1644390us) ts_tablet_manager.cc:1145] Deleting tablet 06edfca60eed4dd3a2dee7b662b0970c
0112 02:52:42.800329 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.800379 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1640677}
W20250112 02:52:42.800904 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 06edfca60eed4dd3a2dee7b662b0970c because the tablet was not found. No further retry: Not found: Tablet not found: 06edfca60eed4dd3a2dee7b662b0970c
I20250112 02:52:42.801187 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 307) took 1631 ms. Trace:
I20250112 02:52:42.801513 2024 rpcz_store.cc:276] 0112 02:52:41.169782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.169847 (+ 65us) service_pool.cc:225] Handling call
0112 02:52:42.801106 (+1631259us) ts_tablet_manager.cc:1145] Deleting tablet a3ffdc237f8640f0b378809f93fc6292
0112 02:52:42.801113 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.801172 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1631123}
I20250112 02:52:42.801190 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
W20250112 02:52:42.801924 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet a3ffdc237f8640f0b378809f93fc6292 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3ffdc237f8640f0b378809f93fc6292 already in progress: creating tablet
I20250112 02:52:42.801959 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:42.802228 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 308) took 1626 ms. Trace:
I20250112 02:52:42.802333 2024 rpcz_store.cc:276] 0112 02:52:41.175995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.176040 (+ 45us) service_pool.cc:225] Handling call
0112 02:52:42.802164 (+1626124us) ts_tablet_manager.cc:1145] Deleting tablet 1a8a178bafcb457f9b0e9587df560493
0112 02:52:42.802172 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.802215 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1600240}
I20250112 02:52:42.802430 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
W20250112 02:52:42.803001 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 1a8a178bafcb457f9b0e9587df560493 because the tablet was not found. No further retry: Not found: Tablet not found: 1a8a178bafcb457f9b0e9587df560493
I20250112 02:52:42.803146 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet d95f98ae7bdf42518bbff55e257fea88 (table default.Table_2fcdbb05970f4a098f7ba3e287fa0348 [id=70c61753e9944e1fbd65e9ec41a48ee2]). Expected version 1 got 0
I20250112 02:52:42.803988 2908 tablet.cc:1721] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:42.804811 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 309) took 1627 ms. Trace:
I20250112 02:52:42.804917 2853 tablet.cc:1721] T b9537fcd96eb41cc8f680075ec71137b P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:42.813766 2908 tablet.cc:1716] T b9537fcd96eb41cc8f680075ec71137b P fe36951a83fb4e57b26e187edbcbfe3d: Skipping requested alter to schema version 3, tablet already version 3
I20250112 02:52:42.815064 1173 tablet_bootstrap.cc:492] T 9ded8bcdda914ea6bc4f9bb92d413dee P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:42.816597 1173 tablet_bootstrap.cc:654] T 9ded8bcdda914ea6bc4f9bb92d413dee P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.818192 2853 tablet.cc:1716] T b9537fcd96eb41cc8f680075ec71137b P f1e775296eb746db8b318f17d26820e6: Skipping requested alter to schema version 3, tablet already version 3
I20250112 02:52:42.819115 1177 tablet_bootstrap.cc:654] T cec5346e13164274ae7bc3b6aa241f61 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.821372 1181 tablet_bootstrap.cc:492] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:42.821527 1181 ts_tablet_manager.cc:1397] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.003s
I20250112 02:52:42.822114 1181 raft_consensus.cc:357] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.824446 1181 raft_consensus.cc:383] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.824545 1181 raft_consensus.cc:738] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.824771 1181 consensus_queue.cc:260] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.825318 1181 ts_tablet_manager.cc:1428] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 02:52:42.826515 2865 tablet.cc:1721] T b9537fcd96eb41cc8f680075ec71137b P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:42.828226 1181 tablet_bootstrap.cc:492] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:42.829133 2914 raft_consensus.cc:491] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 436220c23fa948efb5b6ca15875d8b0e)
I20250112 02:52:42.829411 2914 raft_consensus.cc:513] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:42.830662 464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd25f03f25df49178a5c53c636ecf304" candidate_term: 5 candidate_status { last_received { term: 4 index: 704 } } ignore_live_leader: false dest_uuid: "436220c23fa948efb5b6ca15875d8b0e" is_pre_election: true
I20250112 02:52:42.840101 2609 raft_consensus.cc:491] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.840210 2609 raft_consensus.cc:513] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.840708 2609 leader_election.cc:290] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.841356 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8429a3d8636d4e2ca4c1191f85b9266b" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:42.841467 989 leader_election.cc:343] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 8429a3d8636d4e2ca4c1191f85b9266b
W20250112 02:52:42.842006 991 leader_election.cc:343] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.842114 991 leader_election.cc:304] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:42.842466 2609 raft_consensus.cc:2743] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:42.843652 1173 tablet_bootstrap.cc:492] T 9ded8bcdda914ea6bc4f9bb92d413dee P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:42.843843 1173 ts_tablet_manager.cc:1397] T 9ded8bcdda914ea6bc4f9bb92d413dee P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.000s
I20250112 02:52:42.845258 1173 raft_consensus.cc:357] T 9ded8bcdda914ea6bc4f9bb92d413dee P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.845530 1173 raft_consensus.cc:383] T 9ded8bcdda914ea6bc4f9bb92d413dee P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.845644 1173 raft_consensus.cc:738] T 9ded8bcdda914ea6bc4f9bb92d413dee P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.845942 1173 consensus_queue.cc:260] T 9ded8bcdda914ea6bc4f9bb92d413dee P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.846808 1173 ts_tablet_manager.cc:1428] T 9ded8bcdda914ea6bc4f9bb92d413dee P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:42.839886 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:42.847941 1181 tablet_bootstrap.cc:654] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.856231 2865 tablet.cc:1716] T b9537fcd96eb41cc8f680075ec71137b P 289afc116aa74408aa5bff74e11c50d8: Skipping requested alter to schema version 3, tablet already version 3
I20250112 02:52:42.857445 1181 tablet_bootstrap.cc:492] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:42.857609 1181 ts_tablet_manager.cc:1397] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.003s
I20250112 02:52:42.858177 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:42.858443 1181 raft_consensus.cc:357] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.858631 1181 raft_consensus.cc:383] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.858651 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:42.858700 1181 raft_consensus.cc:738] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.858916 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:42.858929 1181 consensus_queue.cc:260] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.859445 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:42.859521 1181 ts_tablet_manager.cc:1428] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 02:52:42.859778 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:42.860064 1181 tablet_bootstrap.cc:492] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:42.860361 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:42.860577 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 4d25815a5df94e099e7481c48c3da793 (table default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c]). Expected version 2 got 0
I20250112 02:52:42.860718 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 17c54c5dc86446cf85974777c8145ef6 (table default.Table_8ab566cf9c8f45eba4dc9bb85264c823 [id=a622f7a973344a388b3275adbcce6234]). Expected version 1 got 0
I20250112 02:52:42.860857 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3e67f0a0c065498f9a644220414415b0 (table default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022]). Expected version 1 got 0
I20250112 02:52:42.861399 1181 tablet_bootstrap.cc:654] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.861481 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:42.861636 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264]). Expected version 1 got 0
I20250112 02:52:42.861840 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:42.862749 1173 tablet_bootstrap.cc:492] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:42.863680 2609 raft_consensus.cc:491] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.863775 2609 raft_consensus.cc:513] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.864220 2609 leader_election.cc:290] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.864559 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea18666ec324403b9f10a099688438b8" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:42.864715 917 raft_consensus.cc:2463] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:42.865053 1173 tablet_bootstrap.cc:654] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.865018 989 leader_election.cc:304] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:42.865229 2609 raft_consensus.cc:2798] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:42.865303 2609 raft_consensus.cc:491] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:42.865365 2609 raft_consensus.cc:3054] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.867571 2609 raft_consensus.cc:513] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.868446 2609 leader_election.cc:290] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.868402 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea18666ec324403b9f10a099688438b8" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
W20250112 02:52:42.868866 991 leader_election.cc:343] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.868861 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea18666ec324403b9f10a099688438b8" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:42.869016 917 raft_consensus.cc:3054] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.871079 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea18666ec324403b9f10a099688438b8" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:42.871506 991 leader_election.cc:343] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.871930 917 raft_consensus.cc:2463] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:42.872382 989 leader_election.cc:304] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:42.872623 2609 raft_consensus.cc:2798] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:42.872725 2609 raft_consensus.cc:695] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:42.872941 2609 consensus_queue.cc:237] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.874889 1177 tablet_bootstrap.cc:492] T cec5346e13164274ae7bc3b6aa241f61 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:42.875010 1177 ts_tablet_manager.cc:1397] T cec5346e13164274ae7bc3b6aa241f61 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.001s
I20250112 02:52:42.875730 1177 raft_consensus.cc:357] T cec5346e13164274ae7bc3b6aa241f61 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.875882 1177 raft_consensus.cc:383] T cec5346e13164274ae7bc3b6aa241f61 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.875922 1177 raft_consensus.cc:738] T cec5346e13164274ae7bc3b6aa241f61 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.876612 1181 tablet_bootstrap.cc:492] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:42.876574 2914 leader_election.cc:290] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 436220c23fa948efb5b6ca15875d8b0e (127.0.104.62:42341), f4a49a5756b143fcb767c3f41463f8d9 (127.0.104.61:38771)
I20250112 02:52:42.876722 1181 ts_tablet_manager.cc:1397] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20250112 02:52:42.877312 1181 raft_consensus.cc:357] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.877466 1181 raft_consensus.cc:383] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.877509 1181 raft_consensus.cc:738] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.877669 1181 consensus_queue.cc:260] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.878258 1181 ts_tablet_manager.cc:1428] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 02:52:42.878530 1181 tablet_bootstrap.cc:492] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:42.885630 1181 tablet_bootstrap.cc:654] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.888741 1177 consensus_queue.cc:260] T cec5346e13164274ae7bc3b6aa241f61 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.889230 1177 ts_tablet_manager.cc:1428] T cec5346e13164274ae7bc3b6aa241f61 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250112 02:52:42.895236 1177 tablet_bootstrap.cc:492] T 2a6650515ea540a8bbf21e2f88363360 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:42.896828 1177 tablet_bootstrap.cc:654] T 2a6650515ea540a8bbf21e2f88363360 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.904525 1181 tablet_bootstrap.cc:492] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:42.904724 1181 ts_tablet_manager.cc:1397] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.006s
I20250112 02:52:42.905287 1181 raft_consensus.cc:357] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.905537 1181 raft_consensus.cc:383] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.905625 1181 raft_consensus.cc:738] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.905885 1181 consensus_queue.cc:260] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.913681 1181 ts_tablet_manager.cc:1428] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
W20250112 02:52:42.914541 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 56ddc025a8314f6f84d229191eb56692 consensus state has no leader
I20250112 02:52:42.914717 446 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 193) took 1623 ms. Trace:
I20250112 02:52:42.914788 446 rpcz_store.cc:276] 0112 02:52:41.291699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.935555 (+643856us) service_pool.cc:225] Handling call
0112 02:52:41.935939 (+ 384us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:41.936108 (+ 169us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:41.936233 (+ 125us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:41.936292 (+ 59us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:41.983930 (+ 47638us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:42.914672 (+930742us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:41.936685 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.936749 (+ 64us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.936753 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:41.936756 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:41.936814 (+ 58us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.936820 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.936823 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:41.936861 (+ 38us) tablet.cc:698] Row locks acquired
0112 02:52:41.936863 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:41.936883 (+ 20us) write_op.cc:270] Start()
0112 02:52:41.936903 (+ 20us) write_op.cc:276] Timestamp: P: 1736650361936807 usec, L: 0
0112 02:52:41.936906 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:41.937022 (+ 116us) log.cc:844] Serialized 405 byte log entry
0112 02:52:41.939692 (+ 2670us) op_driver.cc:464] REPLICATION: finished
0112 02:52:41.940129 (+ 437us) write_op.cc:301] APPLY: starting
0112 02:52:41.940138 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:41.940160 (+ 22us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:41.940178 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:41.940211 (+ 33us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:41.940242 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:41.940252 (+ 10us) write_op.cc:312] APPLY: finished
0112 02:52:41.940306 (+ 54us) log.cc:844] Serialized 23 byte log entry
0112 02:52:41.940345 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.940359 (+ 14us) write_op.cc:454] Released schema lock
0112 02:52:41.940365 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:41.940432 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.940437 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":410,"apply.run_cpu_time_us":291,"apply.run_wall_time_us":291,"num_ops":1,"prepare.queue_time_us":37,"prepare.run_cpu_time_us":390,"prepare.run_wall_time_us":389,"replication_time_us":2774}]]}
I20250112 02:52:42.922612 1181 tablet_bootstrap.cc:492] T 3d418cda9ed5449fa45690130cd6d5bc P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:42.928565 1181 tablet_bootstrap.cc:654] T 3d418cda9ed5449fa45690130cd6d5bc P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.954488 1173 tablet_bootstrap.cc:492] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:42.954628 1173 ts_tablet_manager.cc:1397] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.092s user 0.005s sys 0.000s
I20250112 02:52:42.961192 2326 raft_consensus.cc:491] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.961344 2326 raft_consensus.cc:513] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.969142 2787 raft_consensus.cc:491] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:42.969286 2787 raft_consensus.cc:513] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:42.969823 2787 leader_election.cc:290] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:42.970432 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efcd32b4ccc4fada5adb3ca3bcdddaa" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:42.970928 855 leader_election.cc:343] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.972746 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efcd32b4ccc4fada5adb3ca3bcdddaa" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:42.973373 854 leader_election.cc:343] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:42.973484 854 leader_election.cc:304] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:42.973887 2673 raft_consensus.cc:2743] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:42.973925 2787 consensus_queue.cc:1035] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:42.974696 1177 tablet_bootstrap.cc:492] T 2a6650515ea540a8bbf21e2f88363360 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:42.974875 1177 ts_tablet_manager.cc:1397] T 2a6650515ea540a8bbf21e2f88363360 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.080s user 0.003s sys 0.000s
I20250112 02:52:42.975638 1177 raft_consensus.cc:357] T 2a6650515ea540a8bbf21e2f88363360 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.975864 1177 raft_consensus.cc:383] T 2a6650515ea540a8bbf21e2f88363360 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.975852 2326 leader_election.cc:290] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.975946 1177 raft_consensus.cc:738] T 2a6650515ea540a8bbf21e2f88363360 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.976126 1177 consensus_queue.cc:260] T 2a6650515ea540a8bbf21e2f88363360 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.976449 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec5346e13164274ae7bc3b6aa241f61" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:42.976593 772 raft_consensus.cc:2463] T cec5346e13164274ae7bc3b6aa241f61 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:42.976966 991 leader_election.cc:304] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:42.977224 2326 raft_consensus.cc:2798] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:42.977293 2326 raft_consensus.cc:491] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:42.977411 2326 raft_consensus.cc:3054] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.977782 1173 raft_consensus.cc:357] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.977926 1173 raft_consensus.cc:383] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:42.977962 1173 raft_consensus.cc:738] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:42.978107 1173 consensus_queue.cc:260] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.978523 1173 ts_tablet_manager.cc:1428] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.024s user 0.001s sys 0.000s
I20250112 02:52:42.978734 1173 tablet_bootstrap.cc:492] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:42.979872 1173 tablet_bootstrap.cc:654] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.979952 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec5346e13164274ae7bc3b6aa241f61" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:42.980094 910 raft_consensus.cc:2463] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:42.980181 2326 raft_consensus.cc:513] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.977232 2024 rpcz_store.cc:276] 0112 02:52:41.176982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.177022 (+ 40us) service_pool.cc:225] Handling call
0112 02:52:42.804731 (+1627709us) ts_tablet_manager.cc:1145] Deleting tablet 7a34d03a80ae402db6607261a5c40b60
0112 02:52:42.804742 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.804795 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1602687}
I20250112 02:52:42.980525 1177 ts_tablet_manager.cc:1428] T 2a6650515ea540a8bbf21e2f88363360 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250112 02:52:42.980690 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 312) took 1759 ms. Trace:
I20250112 02:52:42.980736 1177 tablet_bootstrap.cc:492] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:42.980758 2024 rpcz_store.cc:276] 0112 02:52:41.221509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.221559 (+ 50us) service_pool.cc:225] Handling call
0112 02:52:42.980594 (+1759035us) ts_tablet_manager.cc:1145] Deleting tablet 13578542e9e741e8817e0ee63b10f1a1
0112 02:52:42.980601 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.980676 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1747450}
I20250112 02:52:42.980996 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 319) took 1693 ms. Trace:
I20250112 02:52:42.981050 2024 rpcz_store.cc:276] 0112 02:52:41.287948 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.288169 (+ 221us) service_pool.cc:225] Handling call
0112 02:52:42.980934 (+1692765us) ts_tablet_manager.cc:1145] Deleting tablet e5c10fad790046a88fbc8ebe0303c481
0112 02:52:42.980941 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.980985 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1692616}
I20250112 02:52:42.981197 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec5346e13164274ae7bc3b6aa241f61" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:42.981283 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 320) took 1693 ms. Trace:
I20250112 02:52:42.981328 772 raft_consensus.cc:3054] T cec5346e13164274ae7bc3b6aa241f61 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.981333 2024 rpcz_store.cc:276] 0112 02:52:41.288031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.288346 (+ 315us) service_pool.cc:225] Handling call
0112 02:52:42.981222 (+1692876us) ts_tablet_manager.cc:1145] Deleting tablet 9163878cfeb64693a9c5493e812546af
0112 02:52:42.981230 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.981273 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1692770}
I20250112 02:52:42.981549 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 321) took 1693 ms. Trace:
I20250112 02:52:42.981600 2024 rpcz_store.cc:276] 0112 02:52:41.288113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.288473 (+ 360us) service_pool.cc:225] Handling call
0112 02:52:42.981492 (+1693019us) ts_tablet_manager.cc:1145] Deleting tablet ea33e5c1a1fb46599c5836a36b130051
0112 02:52:42.981499 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.981538 (+ 39us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1692899}
I20250112 02:52:42.981817 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 325) took 1693 ms. Trace:
I20250112 02:52:42.981890 1177 tablet_bootstrap.cc:654] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:42.981874 2024 rpcz_store.cc:276] 0112 02:52:41.288588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.288693 (+ 105us) service_pool.cc:225] Handling call
0112 02:52:42.981750 (+1693057us) ts_tablet_manager.cc:1145] Deleting tablet 8ee98f055024429eafaa29bb961cab87
0112 02:52:42.981757 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.981804 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1692925}
I20250112 02:52:42.982138 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 330) took 1693 ms. Trace:
I20250112 02:52:42.982196 2024 rpcz_store.cc:276] 0112 02:52:41.289033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.289076 (+ 43us) service_pool.cc:225] Handling call
0112 02:52:42.982059 (+1692983us) ts_tablet_manager.cc:1145] Deleting tablet d45fce86e02f46d4a57af718fa2a62c8
0112 02:52:42.982082 (+ 23us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.982127 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1692837}
I20250112 02:52:42.982458 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 332) took 1693 ms. Trace:
I20250112 02:52:42.982523 2024 rpcz_store.cc:276] 0112 02:52:41.289183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.289244 (+ 61us) service_pool.cc:225] Handling call
0112 02:52:42.982387 (+1693143us) ts_tablet_manager.cc:1145] Deleting tablet 2730634a5cfd4c2fb5c4e30100c556d4
0112 02:52:42.982394 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.982443 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1693030}
I20250112 02:52:42.982758 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 334) took 1693 ms. Trace:
I20250112 02:52:42.982811 2024 rpcz_store.cc:276] 0112 02:52:41.289318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.289377 (+ 59us) service_pool.cc:225] Handling call
0112 02:52:42.982688 (+1693311us) ts_tablet_manager.cc:1145] Deleting tablet b9c006281dc949119951c74e23167d7f
0112 02:52:42.982695 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.982748 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1693193}
I20250112 02:52:42.983084 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 329) took 1694 ms. Trace:
I20250112 02:52:42.983139 2024 rpcz_store.cc:276] 0112 02:52:41.288962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.289510 (+ 548us) service_pool.cc:225] Handling call
0112 02:52:42.982966 (+1693456us) ts_tablet_manager.cc:1145] Deleting tablet 73136be25c79491784d3ea71aa9b27f2
0112 02:52:42.982973 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.983073 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1693324}
I20250112 02:52:42.983321 2024 tablet_replica.cc:331] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:42.983485 2024 raft_consensus.cc:2238] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:42.983600 2024 raft_consensus.cc:2267] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:42.984020 2326 leader_election.cc:290] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:42.984124 2024 ts_tablet_manager.cc:1905] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:42.985200 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 7a34d03a80ae402db6607261a5c40b60 because the tablet was not found. No further retry: Not found: Tablet not found: 7a34d03a80ae402db6607261a5c40b60
W20250112 02:52:42.985451 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 13578542e9e741e8817e0ee63b10f1a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13578542e9e741e8817e0ee63b10f1a1 already in progress: creating tablet
W20250112 02:52:42.985631 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet e5c10fad790046a88fbc8ebe0303c481 because the tablet was not found. No further retry: Not found: Tablet not found: e5c10fad790046a88fbc8ebe0303c481
W20250112 02:52:42.985785 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 9163878cfeb64693a9c5493e812546af because the tablet was not found. No further retry: Not found: Tablet not found: 9163878cfeb64693a9c5493e812546af
W20250112 02:52:42.985962 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet ea33e5c1a1fb46599c5836a36b130051 because the tablet was not found. No further retry: Not found: Tablet not found: ea33e5c1a1fb46599c5836a36b130051
W20250112 02:52:42.986152 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 8ee98f055024429eafaa29bb961cab87 because the tablet was not found. No further retry: Not found: Tablet not found: 8ee98f055024429eafaa29bb961cab87
W20250112 02:52:42.986335 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet d45fce86e02f46d4a57af718fa2a62c8 because the tablet was not found. No further retry: Not found: Tablet not found: d45fce86e02f46d4a57af718fa2a62c8
W20250112 02:52:42.986513 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 2730634a5cfd4c2fb5c4e30100c556d4 because the tablet was not found. No further retry: Not found: Tablet not found: 2730634a5cfd4c2fb5c4e30100c556d4
W20250112 02:52:42.986692 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b9c006281dc949119951c74e23167d7f because the tablet was not found. No further retry: Not found: Tablet not found: b9c006281dc949119951c74e23167d7f
W20250112 02:52:42.986884 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 73136be25c79491784d3ea71aa9b27f2 because the tablet was not found. No further retry: Not found: Tablet not found: 73136be25c79491784d3ea71aa9b27f2
I20250112 02:52:42.988754 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec5346e13164274ae7bc3b6aa241f61" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:42.988904 919 raft_consensus.cc:3054] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:42.990056 2024 ts_tablet_manager.cc:1918] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:42.990139 2024 log.cc:1198] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/8429a3d8636d4e2ca4c1191f85b9266b
I20250112 02:52:42.990360 2024 ts_tablet_manager.cc:1939] T 8429a3d8636d4e2ca4c1191f85b9266b P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:42.990895 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 336) took 1701 ms. Trace:
I20250112 02:52:42.990955 2024 rpcz_store.cc:276] 0112 02:52:41.289459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.289538 (+ 79us) service_pool.cc:225] Handling call
0112 02:52:42.983289 (+1693751us) ts_tablet_manager.cc:1145] Deleting tablet 8429a3d8636d4e2ca4c1191f85b9266b
0112 02:52:42.983296 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.986865 (+ 3569us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.989993 (+ 3128us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:42.990881 (+ 888us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1693626}
I20250112 02:52:42.991233 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 324) took 1702 ms. Trace:
I20250112 02:52:42.991284 2024 rpcz_store.cc:276] 0112 02:52:41.288503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.289690 (+ 1187us) service_pool.cc:225] Handling call
0112 02:52:42.991171 (+1701481us) ts_tablet_manager.cc:1145] Deleting tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69
0112 02:52:42.991178 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:42.991221 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1701391}
I20250112 02:52:42.993286 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 8429a3d8636d4e2ca4c1191f85b9266b (table default.Table_328664e3c01f483ca8e5ff50a497f68e [id=1f5c5f9f5e664250898a3935787b6b51]) successfully deleted
W20250112 02:52:42.993562 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 2fdeaf5fa8cc4a2b97b2178d8ecbad69 because the tablet was not found. No further retry: Not found: Tablet not found: 2fdeaf5fa8cc4a2b97b2178d8ecbad69
I20250112 02:52:42.993981 919 raft_consensus.cc:2463] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:42.994376 989 leader_election.cc:304] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:42.994659 2745 raft_consensus.cc:2798] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:42.994764 2745 raft_consensus.cc:695] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:42.994956 2745 consensus_queue.cc:237] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:42.996141 772 raft_consensus.cc:2463] T cec5346e13164274ae7bc3b6aa241f61 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:43.010080 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd25f03f25df49178a5c53c636ecf304" candidate_term: 5 candidate_status { last_received { term: 4 index: 704 } } ignore_live_leader: false dest_uuid: "f4a49a5756b143fcb767c3f41463f8d9" is_pre_election: true
I20250112 02:52:43.010932 1345 leader_election.cc:304] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd25f03f25df49178a5c53c636ecf304; no voters: 436220c23fa948efb5b6ca15875d8b0e, f4a49a5756b143fcb767c3f41463f8d9
I20250112 02:52:43.011369 2914 raft_consensus.cc:2743] T 00000000000000000000000000000000 P dd25f03f25df49178a5c53c636ecf304 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250112 02:52:43.012382 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 327) took 1723 ms. Trace:
I20250112 02:52:43.012459 2024 rpcz_store.cc:276] 0112 02:52:41.288790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.289377 (+ 587us) service_pool.cc:225] Handling call
0112 02:52:42.991429 (+1702052us) ts_tablet_manager.cc:1145] Deleting tablet 352c87f1dc724e7b94bc2e64abfce8d7
0112 02:52:42.991436 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.012348 (+ 20912us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":315,"tablet-delete.queue_time_us":1701933}
W20250112 02:52:43.013314 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 352c87f1dc724e7b94bc2e64abfce8d7 because the tablet was not found. No further retry: Not found: Tablet not found: 352c87f1dc724e7b94bc2e64abfce8d7
I20250112 02:52:43.017952 2876 raft_consensus.cc:491] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.018080 2876 raft_consensus.cc:513] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.018098 2445 consensus_queue.cc:1035] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.018579 2876 leader_election.cc:290] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.019019 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d5799b5b36487da42dc43b1c70ae26" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:43.019423 855 leader_election.cc:343] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.020210 2445 consensus_queue.cc:1035] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.020371 2876 consensus_queue.cc:1035] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.025949 2876 raft_consensus.cc:491] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.026047 2876 raft_consensus.cc:513] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.026494 2876 leader_election.cc:290] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.026947 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea33053c62dc4707b759d5c1756ab4bd" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:43.027361 855 leader_election.cc:343] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.013213 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 335) took 1723 ms. Trace:
I20250112 02:52:43.025396 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d5799b5b36487da42dc43b1c70ae26" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:43.028105 2024 rpcz_store.cc:276] 0112 02:52:41.289387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.289807 (+ 420us) service_pool.cc:225] Handling call
0112 02:52:43.013125 (+1723318us) ts_tablet_manager.cc:1145] Deleting tablet 681a930541a84f0197ecafe670278e70
0112 02:52:43.013134 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.013201 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1723217}
I20250112 02:52:43.028254 1053 raft_consensus.cc:2463] T 41d5799b5b36487da42dc43b1c70ae26 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:43.028370 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 326) took 1739 ms. Trace:
I20250112 02:52:43.028426 2024 rpcz_store.cc:276] 0112 02:52:41.288694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.289840 (+ 1146us) service_pool.cc:225] Handling call
0112 02:52:43.028304 (+1738464us) ts_tablet_manager.cc:1145] Deleting tablet 4b599e983ec04f53a440d1ab828d12ab
0112 02:52:43.028311 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.028358 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1738360}
I20250112 02:52:43.027966 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea33053c62dc4707b759d5c1756ab4bd" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:43.029105 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 681a930541a84f0197ecafe670278e70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 681a930541a84f0197ecafe670278e70 already in progress: creating tablet
W20250112 02:52:43.029361 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 4b599e983ec04f53a440d1ab828d12ab because the tablet was not found. No further retry: Not found: Tablet not found: 4b599e983ec04f53a440d1ab828d12ab
I20250112 02:52:43.029588 854 leader_election.cc:304] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
W20250112 02:52:43.029891 854 leader_election.cc:343] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.029963 854 leader_election.cc:304] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:43.030144 2673 raft_consensus.cc:2798] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:43.030200 2673 raft_consensus.cc:491] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:43.030261 2673 raft_consensus.cc:3054] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.031025 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 333) took 1741 ms. Trace:
I20250112 02:52:43.031107 2024 rpcz_store.cc:276] 0112 02:52:41.289253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.289927 (+ 674us) service_pool.cc:225] Handling call
0112 02:52:43.028591 (+1738664us) ts_tablet_manager.cc:1145] Deleting tablet b61f16f3d601436bb4b32344de28c1a7
0112 02:52:43.030947 (+ 2356us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.031008 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1738553}
I20250112 02:52:43.031400 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 328) took 1742 ms. Trace:
I20250112 02:52:43.031481 2024 rpcz_store.cc:276] 0112 02:52:41.288878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.289960 (+ 1082us) service_pool.cc:225] Handling call
0112 02:52:43.031329 (+1741369us) ts_tablet_manager.cc:1145] Deleting tablet d4abf59d96414e5faa7cf75ba133832c
0112 02:52:43.031338 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.031387 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1741280}
I20250112 02:52:43.031780 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 313) took 1744 ms. Trace:
I20250112 02:52:43.031842 2024 rpcz_store.cc:276] 0112 02:52:41.287372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.290017 (+ 2645us) service_pool.cc:225] Handling call
0112 02:52:43.031682 (+1741665us) ts_tablet_manager.cc:1145] Deleting tablet cbe30f9860e2492b90e8cedf8ff9daba
0112 02:52:43.031691 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.031766 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1741529}
I20250112 02:52:43.032091 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 323) took 1743 ms. Trace:
I20250112 02:52:43.032135 2024 rpcz_store.cc:276] 0112 02:52:41.288410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.290066 (+ 1656us) service_pool.cc:225] Handling call
0112 02:52:43.032034 (+1741968us) ts_tablet_manager.cc:1145] Deleting tablet 8a12b225131b464494a9f02133d37aea
0112 02:52:43.032043 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.032082 (+ 39us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1741864}
I20250112 02:52:43.032308 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 315) took 1744 ms. Trace:
I20250112 02:52:43.032351 2024 rpcz_store.cc:276] 0112 02:52:41.287575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.290181 (+ 2606us) service_pool.cc:225] Handling call
0112 02:52:43.032265 (+1742084us) ts_tablet_manager.cc:1145] Deleting tablet f05c8426ea3d4fe89fbe6c88c688019f
0112 02:52:43.032271 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.032300 (+ 29us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1741955}
I20250112 02:52:43.032615 2673 raft_consensus.cc:513] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.033354 2673 leader_election.cc:290] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.033524 2673 raft_consensus.cc:2743] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.035007 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d5799b5b36487da42dc43b1c70ae26" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
W20250112 02:52:43.035537 855 leader_election.cc:343] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.035475 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d5799b5b36487da42dc43b1c70ae26" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:43.035669 1049 raft_consensus.cc:3054] T 41d5799b5b36487da42dc43b1c70ae26 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.038322 1049 raft_consensus.cc:2463] T 41d5799b5b36487da42dc43b1c70ae26 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
W20250112 02:52:43.038853 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b61f16f3d601436bb4b32344de28c1a7 because the tablet was not found. No further retry: Not found: Tablet not found: b61f16f3d601436bb4b32344de28c1a7
W20250112 02:52:43.039099 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet d4abf59d96414e5faa7cf75ba133832c because the tablet was not found. No further retry: Not found: Tablet not found: d4abf59d96414e5faa7cf75ba133832c
W20250112 02:52:43.039280 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet cbe30f9860e2492b90e8cedf8ff9daba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbe30f9860e2492b90e8cedf8ff9daba already in progress: creating tablet
W20250112 02:52:43.039486 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 8a12b225131b464494a9f02133d37aea because the tablet was not found. No further retry: Not found: Tablet not found: 8a12b225131b464494a9f02133d37aea
W20250112 02:52:43.039664 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f05c8426ea3d4fe89fbe6c88c688019f because the tablet was not found. No further retry: Not found: Tablet not found: f05c8426ea3d4fe89fbe6c88c688019f
I20250112 02:52:43.039916 854 leader_election.cc:304] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:43.040253 2876 raft_consensus.cc:2798] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:43.040364 2876 raft_consensus.cc:695] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:43.040551 2876 consensus_queue.cc:237] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.047652 2445 consensus_queue.cc:1035] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.049703 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 322) took 1761 ms. Trace:
I20250112 02:52:43.049803 2024 rpcz_store.cc:276] 0112 02:52:41.288238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.290190 (+ 1952us) service_pool.cc:225] Handling call
0112 02:52:43.032518 (+1742328us) ts_tablet_manager.cc:1145] Deleting tablet bb217afc08304c0ea35dee3120347ba1
0112 02:52:43.049624 (+ 17106us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.049687 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1742192}
W20250112 02:52:43.050249 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet bb217afc08304c0ea35dee3120347ba1 because the tablet was not found. No further retry: Not found: Tablet not found: bb217afc08304c0ea35dee3120347ba1
I20250112 02:52:43.051229 1181 tablet_bootstrap.cc:492] T 3d418cda9ed5449fa45690130cd6d5bc P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:43.053359 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 316) took 1765 ms. Trace:
I20250112 02:52:43.053467 2024 rpcz_store.cc:276] 0112 02:52:41.287658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.290337 (+ 2679us) service_pool.cc:225] Handling call
0112 02:52:43.050063 (+1759726us) ts_tablet_manager.cc:1145] Deleting tablet daef3a58605e40978f7389698b22265d
0112 02:52:43.053260 (+ 3197us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.053338 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1759627}
I20250112 02:52:43.053910 1181 ts_tablet_manager.cc:1397] T 3d418cda9ed5449fa45690130cd6d5bc P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.131s user 0.002s sys 0.007s
W20250112 02:52:43.053932 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet daef3a58605e40978f7389698b22265d because the tablet was not found. No further retry: Not found: Tablet not found: daef3a58605e40978f7389698b22265d
I20250112 02:52:43.061600 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 317) took 1773 ms. Trace:
I20250112 02:52:43.061693 2024 rpcz_store.cc:276] 0112 02:52:41.287731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.290454 (+ 2723us) service_pool.cc:225] Handling call
0112 02:52:43.053758 (+1763304us) ts_tablet_manager.cc:1145] Deleting tablet a12c5d58e6c246c983e8a533906ce77d
0112 02:52:43.061512 (+ 7754us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.061583 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1763189}
W20250112 02:52:43.062281 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet a12c5d58e6c246c983e8a533906ce77d because the tablet was not found. No further retry: Not found: Tablet not found: a12c5d58e6c246c983e8a533906ce77d
W20250112 02:52:43.063560 713 consensus_peers.cc:487] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1d9fae290ec4281839991f9d40b105c. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:43.067160 2445 consensus_queue.cc:1035] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.068282 1181 raft_consensus.cc:357] T 3d418cda9ed5449fa45690130cd6d5bc P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.073176 1181 raft_consensus.cc:383] T 3d418cda9ed5449fa45690130cd6d5bc P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.073338 1181 raft_consensus.cc:738] T 3d418cda9ed5449fa45690130cd6d5bc P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.073374 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 331) took 1784 ms. Trace:
I20250112 02:52:43.073470 2024 rpcz_store.cc:276] 0112 02:52:41.289110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.290585 (+ 1475us) service_pool.cc:225] Handling call
0112 02:52:43.061930 (+1771345us) ts_tablet_manager.cc:1145] Deleting tablet 003a3b4705e943c6a900762ec11ddd47
0112 02:52:43.073278 (+ 11348us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.073353 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1771247}
W20250112 02:52:43.073966 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 003a3b4705e943c6a900762ec11ddd47 because the tablet was not found. No further retry: Not found: Tablet not found: 003a3b4705e943c6a900762ec11ddd47
I20250112 02:52:43.075665 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 314) took 1788 ms. Trace:
I20250112 02:52:43.075812 2024 rpcz_store.cc:276] 0112 02:52:41.287477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.292369 (+ 4892us) service_pool.cc:225] Handling call
0112 02:52:43.073787 (+1781418us) ts_tablet_manager.cc:1145] Deleting tablet 83c04f1a2cb04950a9f6170e252c19b4
0112 02:52:43.075557 (+ 1770us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.075646 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1781274}
I20250112 02:52:43.076133 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 341) took 1654 ms. Trace:
I20250112 02:52:43.076210 2024 rpcz_store.cc:276] 0112 02:52:41.421380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.421520 (+ 140us) service_pool.cc:225] Handling call
0112 02:52:43.076060 (+1654540us) ts_tablet_manager.cc:1145] Deleting tablet 70715db0394d46d9b6b289160b00be7d
0112 02:52:43.076071 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.076119 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1654354}
W20250112 02:52:43.076432 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 83c04f1a2cb04950a9f6170e252c19b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83c04f1a2cb04950a9f6170e252c19b4 already in progress: creating tablet
I20250112 02:52:43.076525 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 343) took 1654 ms. Trace:
I20250112 02:52:43.076598 2024 rpcz_store.cc:276] 0112 02:52:41.421637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.421749 (+ 112us) service_pool.cc:225] Handling call
0112 02:52:43.076434 (+1654685us) ts_tablet_manager.cc:1145] Deleting tablet 76009bac22ce4136b148b493a004a0f7
0112 02:52:43.076443 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.076513 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1654548}
I20250112 02:52:43.073693 1181 consensus_queue.cc:260] T 3d418cda9ed5449fa45690130cd6d5bc P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.076881 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 337) took 1655 ms. Trace:
W20250112 02:52:43.076996 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 70715db0394d46d9b6b289160b00be7d because the tablet was not found. No further retry: Not found: Tablet not found: 70715db0394d46d9b6b289160b00be7d
I20250112 02:52:43.076941 2024 rpcz_store.cc:276] 0112 02:52:41.420976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.422701 (+ 1725us) service_pool.cc:225] Handling call
0112 02:52:43.076798 (+1654097us) ts_tablet_manager.cc:1145] Deleting tablet 54215c55966a46e1ab96e352c4e3764e
0112 02:52:43.076807 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.076870 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1653947}
I20250112 02:52:43.077270 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 338) took 1656 ms. Trace:
W20250112 02:52:43.077338 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 76009bac22ce4136b148b493a004a0f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76009bac22ce4136b148b493a004a0f7 already in progress: creating tablet
I20250112 02:52:43.077353 2024 rpcz_store.cc:276] 0112 02:52:41.421086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.422895 (+ 1809us) service_pool.cc:225] Handling call
0112 02:52:43.077184 (+1654289us) ts_tablet_manager.cc:1145] Deleting tablet d63d411f906b4d4eb68343bdcc493e05
0112 02:52:43.077193 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.077257 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1654145}
I20250112 02:52:43.077636 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 339) took 1656 ms. Trace:
I20250112 02:52:43.077708 2024 rpcz_store.cc:276] 0112 02:52:41.421177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.423075 (+ 1898us) service_pool.cc:225] Handling call
0112 02:52:43.077555 (+1654480us) ts_tablet_manager.cc:1145] Deleting tablet 7e1954e7859f422f9690fe4a9384a8bb
0112 02:52:43.077563 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.077624 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1654345}
I20250112 02:52:43.077975 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 340) took 1656 ms. Trace:
I20250112 02:52:43.078063 2024 rpcz_store.cc:276] 0112 02:52:41.421269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.423237 (+ 1968us) service_pool.cc:225] Handling call
0112 02:52:43.077912 (+1654675us) ts_tablet_manager.cc:1145] Deleting tablet 416fb6820cd54e118950c68d381f51b2
0112 02:52:43.077921 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.077962 (+ 41us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1654502}
I20250112 02:52:43.078377 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 342) took 1656 ms. Trace:
I20250112 02:52:43.078447 2024 rpcz_store.cc:276] 0112 02:52:41.421499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.423438 (+ 1939us) service_pool.cc:225] Handling call
0112 02:52:43.078288 (+1654850us) ts_tablet_manager.cc:1145] Deleting tablet 9af55dab202444358ea1a0674d37946d
0112 02:52:43.078298 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.078352 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1654688}
W20250112 02:52:43.078511 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 54215c55966a46e1ab96e352c4e3764e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54215c55966a46e1ab96e352c4e3764e already in progress: creating tablet
I20250112 02:52:43.078735 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 344) took 1628 ms. Trace:
I20250112 02:52:43.078805 2024 rpcz_store.cc:276] 0112 02:52:41.449949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.450002 (+ 53us) service_pool.cc:225] Handling call
0112 02:52:43.078651 (+1628649us) ts_tablet_manager.cc:1145] Deleting tablet 940affdc2c484b7d9fd2de17f9d1cea7
0112 02:52:43.078660 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.078723 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1628446}
W20250112 02:52:43.078913 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet d63d411f906b4d4eb68343bdcc493e05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d63d411f906b4d4eb68343bdcc493e05 already in progress: creating tablet
I20250112 02:52:43.079152 1181 ts_tablet_manager.cc:1428] T 3d418cda9ed5449fa45690130cd6d5bc P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
W20250112 02:52:43.079233 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 7e1954e7859f422f9690fe4a9384a8bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e1954e7859f422f9690fe4a9384a8bb already in progress: creating tablet
I20250112 02:52:43.079533 2024 tablet_replica.cc:331] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:43.079748 2024 raft_consensus.cc:2238] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250112 02:52:43.079764 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 416fb6820cd54e118950c68d381f51b2 because the tablet was not found. No further retry: Not found: Tablet not found: 416fb6820cd54e118950c68d381f51b2
I20250112 02:52:43.079885 2024 raft_consensus.cc:2267] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:43.080099 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 9af55dab202444358ea1a0674d37946d because the tablet was not found. No further retry: Not found: Tablet not found: 9af55dab202444358ea1a0674d37946d
I20250112 02:52:43.080369 2024 ts_tablet_manager.cc:1905] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:43.080366 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 940affdc2c484b7d9fd2de17f9d1cea7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 940affdc2c484b7d9fd2de17f9d1cea7 already in progress: creating tablet
I20250112 02:52:43.080590 1181 tablet_bootstrap.cc:492] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:43.094856 1181 tablet_bootstrap.cc:654] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.115588 2876 raft_consensus.cc:491] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.115785 2876 raft_consensus.cc:513] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.116832 2876 leader_election.cc:290] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.117374 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e7882e07784bdda18dc1ffd20963c8" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:43.117435 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e7882e07784bdda18dc1ffd20963c8" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:43.117878 855 leader_election.cc:343] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:43.118211 854 leader_election.cc:343] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.118295 854 leader_election.cc:304] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:43.118625 2876 raft_consensus.cc:2743] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.121258 2865 tablet.cc:1721] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:43.122175 2908 tablet.cc:1721] T b8b64e4e64cc462891f1ee7a594d5026 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:43.125166 1181 tablet_bootstrap.cc:492] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:43.125344 1181 ts_tablet_manager.cc:1397] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.008s
I20250112 02:52:43.126030 1181 raft_consensus.cc:357] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.126237 1181 raft_consensus.cc:383] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.126324 1181 raft_consensus.cc:738] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.126549 1181 consensus_queue.cc:260] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.135357 1181 ts_tablet_manager.cc:1428] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250112 02:52:43.138753 1181 tablet_bootstrap.cc:492] T 781a9943f705477cb6eae16df4468d2f P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:43.139935 2024 ts_tablet_manager.cc:1918] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:43.140034 2024 log.cc:1198] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/b124c15736084e6d8f69b4fb7c41cf39
I20250112 02:52:43.140290 2024 ts_tablet_manager.cc:1939] T b124c15736084e6d8f69b4fb7c41cf39 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:43.140954 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 347) took 1210 ms. Trace:
I20250112 02:52:43.141047 2024 rpcz_store.cc:276] 0112 02:52:41.930382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.930437 (+ 55us) service_pool.cc:225] Handling call
0112 02:52:43.079496 (+1149059us) ts_tablet_manager.cc:1145] Deleting tablet b124c15736084e6d8f69b4fb7c41cf39
0112 02:52:43.079502 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.104182 (+ 24680us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:43.139854 (+ 35672us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:43.140938 (+ 1084us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1148880}
I20250112 02:52:43.141309 2024 tablet_replica.cc:331] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:43.141422 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet b124c15736084e6d8f69b4fb7c41cf39 (table default.Table_f0fb142b9aab490bb067d3cf64dc47b1 [id=1991c0821ef4478eb0eb9282ab772c0c]) successfully deleted
I20250112 02:52:43.141485 2024 raft_consensus.cc:2238] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:43.141757 2024 raft_consensus.cc:2267] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:43.142179 2024 ts_tablet_manager.cc:1905] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:43.143055 1181 tablet_bootstrap.cc:654] T 781a9943f705477cb6eae16df4468d2f P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.145247 1173 tablet_bootstrap.cc:492] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:43.145349 1173 ts_tablet_manager.cc:1397] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.167s user 0.007s sys 0.013s
I20250112 02:52:43.145937 1173 raft_consensus.cc:357] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.146108 1173 raft_consensus.cc:383] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.146167 1173 raft_consensus.cc:738] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.146327 1173 consensus_queue.cc:260] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.146826 1173 ts_tablet_manager.cc:1428] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 02:52:43.147078 1173 tablet_bootstrap.cc:492] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:43.150856 1173 tablet_bootstrap.cc:654] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.152593 1173 tablet_bootstrap.cc:492] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:43.152755 1173 ts_tablet_manager.cc:1397] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250112 02:52:43.153554 1173 raft_consensus.cc:357] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.153789 1173 raft_consensus.cc:383] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.153895 1173 raft_consensus.cc:738] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.154121 1173 consensus_queue.cc:260] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.154752 1173 ts_tablet_manager.cc:1428] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:43.154974 1173 tablet_bootstrap.cc:492] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:43.155570 1181 tablet_bootstrap.cc:492] T 781a9943f705477cb6eae16df4468d2f P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:43.155706 1181 ts_tablet_manager.cc:1397] T 781a9943f705477cb6eae16df4468d2f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250112 02:52:43.156266 1181 raft_consensus.cc:357] T 781a9943f705477cb6eae16df4468d2f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.156476 1181 raft_consensus.cc:383] T 781a9943f705477cb6eae16df4468d2f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.156553 1181 raft_consensus.cc:738] T 781a9943f705477cb6eae16df4468d2f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.156752 1181 consensus_queue.cc:260] T 781a9943f705477cb6eae16df4468d2f P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.157956 2179 raft_consensus.cc:491] T 2a6650515ea540a8bbf21e2f88363360 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.158071 2179 raft_consensus.cc:513] T 2a6650515ea540a8bbf21e2f88363360 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.158664 2179 leader_election.cc:290] T 2a6650515ea540a8bbf21e2f88363360 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:43.159180 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6650515ea540a8bbf21e2f88363360" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:43.159513 910 raft_consensus.cc:2388] T 2a6650515ea540a8bbf21e2f88363360 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 289afc116aa74408aa5bff74e11c50d8 in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
I20250112 02:52:43.160442 1181 ts_tablet_manager.cc:1428] T 781a9943f705477cb6eae16df4468d2f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250112 02:52:43.160668 1181 tablet_bootstrap.cc:492] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:43.161960 1181 tablet_bootstrap.cc:654] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.158957 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6650515ea540a8bbf21e2f88363360" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:43.165318 714 leader_election.cc:304] T 2a6650515ea540a8bbf21e2f88363360 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:43.167353 2609 raft_consensus.cc:491] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.167460 2609 raft_consensus.cc:513] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.167929 2609 leader_election.cc:290] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:43.168558 1173 tablet_bootstrap.cc:654] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:43.168851 989 leader_election.cc:343] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 7e1954e7859f422f9690fe4a9384a8bb
I20250112 02:52:43.169121 2894 raft_consensus.cc:3054] T 2a6650515ea540a8bbf21e2f88363360 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.169661 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e1954e7859f422f9690fe4a9384a8bb" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:43.170058 991 leader_election.cc:343] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.170159 991 leader_election.cc:304] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:43.170456 2609 raft_consensus.cc:2743] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.171989 2894 raft_consensus.cc:2743] T 2a6650515ea540a8bbf21e2f88363360 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.174571 2024 ts_tablet_manager.cc:1918] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:43.174762 1177 tablet_bootstrap.cc:492] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:43.174945 1177 ts_tablet_manager.cc:1397] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.194s user 0.030s sys 0.025s
I20250112 02:52:43.175607 1177 raft_consensus.cc:357] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.175823 1177 raft_consensus.cc:383] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.175840 2024 log.cc:1198] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/795ac7eaf81d48bb926865d9ebb388b6
I20250112 02:52:43.175917 1177 raft_consensus.cc:738] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.176323 1177 consensus_queue.cc:260] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.176590 1173 tablet_bootstrap.cc:492] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:43.176685 1173 ts_tablet_manager.cc:1397] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20250112 02:52:43.176980 1177 ts_tablet_manager.cc:1428] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:43.177224 2024 ts_tablet_manager.cc:1939] T 795ac7eaf81d48bb926865d9ebb388b6 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:43.177232 1177 tablet_bootstrap.cc:492] T 3d662e0ce13f447595e5f92b94de3018 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:43.177244 1173 raft_consensus.cc:357] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.177392 1173 raft_consensus.cc:383] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.177441 1173 raft_consensus.cc:738] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.177584 1173 consensus_queue.cc:260] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.177848 2024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 348) took 1239 ms. Trace:
I20250112 02:52:43.177958 1173 ts_tablet_manager.cc:1428] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:43.178635 1177 tablet_bootstrap.cc:654] T 3d662e0ce13f447595e5f92b94de3018 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.177918 2024 rpcz_store.cc:276] 0112 02:52:41.937900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.937958 (+ 58us) service_pool.cc:225] Handling call
0112 02:52:43.141272 (+1203314us) ts_tablet_manager.cc:1145] Deleting tablet 795ac7eaf81d48bb926865d9ebb388b6
0112 02:52:43.141280 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:43.149870 (+ 8590us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:43.174485 (+ 24615us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:43.177835 (+ 3350us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1203133}
I20250112 02:52:43.183457 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 795ac7eaf81d48bb926865d9ebb388b6 (table default.Table_178713b464bb4f16970851646a710574 [id=d3f31247f0e848f8b2dfa18f3437ca63]) successfully deleted
I20250112 02:52:43.183773 1173 tablet_bootstrap.cc:492] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:43.184861 1173 tablet_bootstrap.cc:654] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.191145 1173 tablet_bootstrap.cc:492] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:43.191613 1173 ts_tablet_manager.cc:1397] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20250112 02:52:43.192242 1173 raft_consensus.cc:357] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.194777 1173 raft_consensus.cc:383] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.194936 1173 raft_consensus.cc:738] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.195428 1173 consensus_queue.cc:260] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.196136 1173 ts_tablet_manager.cc:1428] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250112 02:52:43.197082 1173 tablet_bootstrap.cc:492] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:43.198460 1173 tablet_bootstrap.cc:654] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.199271 2853 tablet.cc:1721] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:43.202095 1177 tablet_bootstrap.cc:492] T 3d662e0ce13f447595e5f92b94de3018 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:43.202301 1177 ts_tablet_manager.cc:1397] T 3d662e0ce13f447595e5f92b94de3018 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20250112 02:52:43.203080 1177 raft_consensus.cc:357] T 3d662e0ce13f447595e5f92b94de3018 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.204129 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:43.204569 446 catalog_manager.cc:5526] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:43.205277 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet cf54f5185a0f4c70b9057309fca14e56 (table default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55]). Expected version 1 got 0
I20250112 02:52:43.205914 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_5d473cabff2e46e0b531fbce28d9d558 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 1 got 0
I20250112 02:52:43.206213 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:43.206581 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_a9907ed8bfe14625977ec98b67e9d599 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 2 got 0
I20250112 02:52:43.206768 1177 raft_consensus.cc:383] T 3d662e0ce13f447595e5f92b94de3018 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.206856 1177 raft_consensus.cc:738] T 3d662e0ce13f447595e5f92b94de3018 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.207126 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet edfd1ebe729d458ea31eb7feff7ee072 (table default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a]). Expected version 1 got 0
I20250112 02:52:43.207054 1177 consensus_queue.cc:260] T 3d662e0ce13f447595e5f92b94de3018 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.207454 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_b9dff6cb7bbd4d0a88de458399b0d2c0 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 2 got 0
I20250112 02:52:43.207662 1177 ts_tablet_manager.cc:1428] T 3d662e0ce13f447595e5f92b94de3018 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250112 02:52:43.207906 1177 tablet_bootstrap.cc:492] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:43.208570 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ca2165362ee541adac724b63f39b1489 (table default.Table_41b7056b6e9240928d0af52aec2e9338 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 2 got 0
I20250112 02:52:43.209167 1177 tablet_bootstrap.cc:654] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.209582 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:43.209826 1173 tablet_bootstrap.cc:492] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:43.209959 1173 ts_tablet_manager.cc:1397] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250112 02:52:43.210479 1173 raft_consensus.cc:357] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.210650 1173 raft_consensus.cc:383] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.210696 1173 raft_consensus.cc:738] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.210919 1173 consensus_queue.cc:260] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.211302 1173 ts_tablet_manager.cc:1428] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 02:52:43.211865 1181 tablet_bootstrap.cc:492] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:43.212028 1181 ts_tablet_manager.cc:1397] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.007s
I20250112 02:52:43.212756 1181 raft_consensus.cc:357] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.212992 1181 raft_consensus.cc:383] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.213106 1181 raft_consensus.cc:738] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.213373 1181 consensus_queue.cc:260] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.217386 1181 ts_tablet_manager.cc:1428] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250112 02:52:43.217681 1181 tablet_bootstrap.cc:492] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:43.219051 1181 tablet_bootstrap.cc:654] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.211357 446 catalog_manager.cc:5526] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:43.224895 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:43.225538 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:43.225724 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:43.225942 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 56ddc025a8314f6f84d229191eb56692 (table default.Table_d22db1cc1d354af495d9888de1bfc22a [id=2ddd5dd296b94bc8b96a97b66e775836]). Expected version 1 got 0
I20250112 02:52:43.227645 1173 tablet_bootstrap.cc:492] T 3d662e0ce13f447595e5f92b94de3018 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:43.229146 1173 tablet_bootstrap.cc:654] T 3d662e0ce13f447595e5f92b94de3018 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.230793 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:43.231073 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3dab0466476c4a9aa265271c1abd91a4 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:43.232192 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:43.232992 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:43.233618 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:43.234305 446 catalog_manager.cc:5526] T 6171b8ff2bfe41d598b0cb2535799a11 P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:43.235404 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:43.235982 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:43.237783 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:43.238148 2876 raft_consensus.cc:491] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.238255 2876 raft_consensus.cc:513] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.238548 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet d95f98ae7bdf42518bbff55e257fea88 (table default.Table_2fcdbb05970f4a098f7ba3e287fa0348 [id=70c61753e9944e1fbd65e9ec41a48ee2]). Expected version 1 got 0
I20250112 02:52:43.238818 2876 leader_election.cc:290] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.238827 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:43.239820 1177 tablet_bootstrap.cc:492] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:43.239984 1177 ts_tablet_manager.cc:1397] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20250112 02:52:43.240557 1177 raft_consensus.cc:357] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.240760 1177 raft_consensus.cc:383] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.240836 1177 raft_consensus.cc:738] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.241041 1177 consensus_queue.cc:260] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.241520 1177 ts_tablet_manager.cc:1428] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 02:52:43.241896 1177 tablet_bootstrap.cc:492] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:43.242127 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c54c5dc86446cf85974777c8145ef6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:43.242383 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:43.242849 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
W20250112 02:52:43.242847 854 leader_election.cc:343] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.243121 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:43.242264 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c54c5dc86446cf85974777c8145ef6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:43.243690 855 leader_election.cc:343] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.243780 855 leader_election.cc:304] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:43.244060 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 777272e7159b438ebff4db93698411d0 (table default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]). Expected version 1 got 0
I20250112 02:52:43.244081 2876 raft_consensus.cc:2743] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.244321 446 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 17c54c5dc86446cf85974777c8145ef6 (table default.Table_8ab566cf9c8f45eba4dc9bb85264c823 [id=a622f7a973344a388b3275adbcce6234]). Expected version 1 got 0
I20250112 02:52:43.245064 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 8a88463991294bd8a335337f8b072c3d
I20250112 02:52:43.245113 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 781a9943f705477cb6eae16df4468d2f
I20250112 02:52:43.245532 454 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40610 (request call id 191) took 1999 ms. Trace:
W20250112 02:52:43.245707 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5524d75599f04becbe6a94426adb7026 consensus state has no leader
W20250112 02:52:43.245833 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37d38f5e525440f8a0f7822620933435 consensus state has no leader
I20250112 02:52:43.245854 447 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 190) took 2002 ms. Trace:
I20250112 02:52:43.245996 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_2add53a7889440009ab2d3b0e58c5381 [id=c1fbaa09536c4fa994c6949a3fd58c75] alter complete (version 2)
I20250112 02:52:43.245968 447 rpcz_store.cc:276] 0112 02:52:41.243738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.926373 (+682635us) service_pool.cc:225] Handling call
0112 02:52:41.926720 (+ 347us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:41.926921 (+ 201us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:41.927118 (+ 197us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:41.927213 (+ 95us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:41.998386 (+ 71173us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:43.245819 (+1247433us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:41.927719 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.927811 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.927820 (+ 9us) write_op.cc:435] Acquired schema lock
0112 02:52:41.927824 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:41.927935 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.927966 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.927971 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:41.928060 (+ 89us) tablet.cc:698] Row locks acquired
0112 02:52:41.928062 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:41.928094 (+ 32us) write_op.cc:270] Start()
0112 02:52:41.928124 (+ 30us) write_op.cc:276] Timestamp: P: 1736650361928016 usec, L: 0
0112 02:52:41.928127 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:41.928300 (+ 173us) log.cc:844] Serialized 405 byte log entry
0112 02:52:41.933653 (+ 5353us) op_driver.cc:464] REPLICATION: finished
0112 02:52:41.933746 (+ 93us) write_op.cc:301] APPLY: starting
0112 02:52:41.933757 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:41.933797 (+ 40us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:41.933835 (+ 38us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:41.933879 (+ 44us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:41.933914 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:41.933921 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:41.933979 (+ 58us) log.cc:844] Serialized 23 byte log entry
0112 02:52:41.934025 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.934038 (+ 13us) write_op.cc:454] Released schema lock
0112 02:52:41.934043 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:41.934098 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.934101 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":54,"apply.run_cpu_time_us":347,"apply.run_wall_time_us":346,"num_ops":1,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":678,"prepare.run_wall_time_us":679,"raft.queue_time_us":391,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":250,"replication_time_us":5502}]]}
I20250112 02:52:43.247840 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a88463991294bd8a335337f8b072c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:43.248052 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 781a9943f705477cb6eae16df4468d2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:43.248217 450 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 192) took 1978 ms. Trace:
I20250112 02:52:43.248306 450 rpcz_store.cc:276] 0112 02:52:41.267785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.933205 (+665420us) service_pool.cc:225] Handling call
0112 02:52:41.933585 (+ 380us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:41.933758 (+ 173us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:41.933907 (+ 149us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:41.933970 (+ 63us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:41.959082 (+ 25112us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:43.245918 (+1286836us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:41.934366 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.934432 (+ 66us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.934436 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:41.934438 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:41.934496 (+ 58us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.934503 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.934506 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:41.934546 (+ 40us) tablet.cc:698] Row locks acquired
0112 02:52:41.934547 (+ 1us) write_op.cc:260] PREPARE: finished
0112 02:52:41.934570 (+ 23us) write_op.cc:270] Start()
0112 02:52:41.934590 (+ 20us) write_op.cc:276] Timestamp: P: 1736650361934493 usec, L: 0
0112 02:52:41.934593 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:41.934695 (+ 102us) log.cc:844] Serialized 405 byte log entry
0112 02:52:41.939637 (+ 4942us) op_driver.cc:464] REPLICATION: finished
0112 02:52:41.939765 (+ 128us) write_op.cc:301] APPLY: starting
0112 02:52:41.939776 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:41.939801 (+ 25us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:41.939832 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:41.939865 (+ 33us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:41.939889 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:41.939895 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:41.939956 (+ 61us) log.cc:844] Serialized 23 byte log entry
0112 02:52:41.940001 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.940011 (+ 10us) write_op.cc:454] Released schema lock
0112 02:52:41.940017 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:41.940076 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.940079 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":93,"apply.run_cpu_time_us":303,"apply.run_wall_time_us":302,"num_ops":1,"prepare.queue_time_us":40,"prepare.run_cpu_time_us":366,"prepare.run_wall_time_us":365,"replication_time_us":5022}]]}
I20250112 02:52:43.249002 2876 raft_consensus.cc:491] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.249121 2876 raft_consensus.cc:513] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.249614 2876 leader_election.cc:290] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.249614 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 23e7a2e88c374e949ced92c557225cbf
I20250112 02:52:43.250093 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a3af0ea4e954053ae89814f3e74e6b7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:43.250140 455 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40610 (request call id 194) took 1952 ms. Trace:
W20250112 02:52:43.250454 855 leader_election.cc:343] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.250249 455 rpcz_store.cc:276] 0112 02:52:41.297220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.944004 (+646784us) service_pool.cc:225] Handling call
0112 02:52:41.944343 (+ 339us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:41.944551 (+ 208us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:52:41.944611 (+ 60us) catalog_manager.cc:2762] Locking tablets
0112 02:52:41.944732 (+ 121us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:52:41.998439 (+ 53707us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.998464 (+ 25us) write_op.cc:454] Released schema lock
0112 02:52:41.998587 (+ 123us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:52:42.003123 (+ 4536us) catalog_manager.cc:2810] Committing in-memory state
0112 02:52:43.249530 (+1246407us) catalog_manager.cc:2814] Aborting table tasks
0112 02:52:43.250094 (+ 564us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:41.945318 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.945402 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.945407 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:41.945409 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:41.945491 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.945501 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.945505 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:41.945590 (+ 85us) tablet.cc:698] Row locks acquired
0112 02:52:41.945593 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:41.945616 (+ 23us) write_op.cc:270] Start()
0112 02:52:41.945636 (+ 20us) write_op.cc:276] Timestamp: P: 1736650361945537 usec, L: 0
0112 02:52:41.945640 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:41.945771 (+ 131us) log.cc:844] Serialized 765 byte log entry
0112 02:52:41.954693 (+ 8922us) op_driver.cc:464] REPLICATION: finished
0112 02:52:41.954829 (+ 136us) write_op.cc:301] APPLY: starting
0112 02:52:41.954841 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:41.954875 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:41.954895 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:41.954946 (+ 51us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:41.954976 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:41.954983 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:41.955051 (+ 68us) log.cc:844] Serialized 29 byte log entry
0112 02:52:41.955104 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.955120 (+ 16us) write_op.cc:454] Released schema lock
0112 02:52:41.955127 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":92,"apply.run_cpu_time_us":346,"apply.run_wall_time_us":346,"num_ops":2,"prepare.queue_time_us":44,"prepare.run_cpu_time_us":525,"prepare.run_wall_time_us":525,"raft.queue_time_us":105,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":194,"replication_time_us":9023}]]}
I20250112 02:52:43.251228 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_3d820aeff9534cc98f5722887aed5300" } new_table_name: "default.Table_6113c9f3ad094f0381176b06d637966c" modify_external_catalogs: true
I20250112 02:52:43.251842 453 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 188) took 2178 ms. Trace:
I20250112 02:52:43.252360 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a88463991294bd8a335337f8b072c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:43.251962 453 rpcz_store.cc:276] 0112 02:52:41.073045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.699240 (+626195us) service_pool.cc:225] Handling call
0112 02:52:41.705476 (+ 6236us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:41.705667 (+ 191us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:41.705803 (+ 136us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:41.705867 (+ 64us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:41.801080 (+ 95213us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:43.251802 (+1450722us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:41.764454 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.764559 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.764565 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:41.764567 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:41.764652 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.764662 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.764667 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:41.764718 (+ 51us) tablet.cc:698] Row locks acquired
0112 02:52:41.764720 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:41.764745 (+ 25us) write_op.cc:270] Start()
0112 02:52:41.764769 (+ 24us) write_op.cc:276] Timestamp: P: 1736650361764703 usec, L: 0
0112 02:52:41.764773 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:41.764910 (+ 137us) log.cc:844] Serialized 405 byte log entry
0112 02:52:41.795812 (+ 30902us) op_driver.cc:464] REPLICATION: finished
0112 02:52:41.796038 (+ 226us) write_op.cc:301] APPLY: starting
0112 02:52:41.796052 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:41.796086 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:41.796135 (+ 49us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:41.796208 (+ 73us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:41.796236 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:41.796242 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:41.796345 (+ 103us) log.cc:844] Serialized 23 byte log entry
0112 02:52:41.796415 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.796429 (+ 14us) write_op.cc:454] Released schema lock
0112 02:52:41.796453 (+ 24us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:41.796519 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.796523 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":151,"apply.run_cpu_time_us":474,"apply.run_wall_time_us":473,"num_ops":1,"prepare.queue_time_us":58175,"prepare.run_cpu_time_us":516,"prepare.run_wall_time_us":515,"replication_time_us":31011}]]}
I20250112 02:52:43.252538 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 781a9943f705477cb6eae16df4468d2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:43.252681 2024 tablet_replica.cc:331] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:43.252830 2024 raft_consensus.cc:2238] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:43.252949 2024 raft_consensus.cc:2267] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:43.252827 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_aca3d1a26d2049e5a0c44926b28dcfcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:43.253398 2024 ts_tablet_manager.cc:1905] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:43.253485 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aca3d1a26d2049e5a0c44926b28dcfcc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:43.256805 714 consensus_peers.cc:487] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1d9fae290ec4281839991f9d40b105c. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:43.258837 1181 tablet_bootstrap.cc:492] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:43.258951 1181 ts_tablet_manager.cc:1397] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.003s
I20250112 02:52:43.259620 1181 raft_consensus.cc:357] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.259794 1181 raft_consensus.cc:383] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.259846 1181 raft_consensus.cc:738] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.261184 1173 tablet_bootstrap.cc:492] T 3d662e0ce13f447595e5f92b94de3018 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:43.261296 1173 ts_tablet_manager.cc:1397] T 3d662e0ce13f447595e5f92b94de3018 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.006s
I20250112 02:52:43.261869 1173 raft_consensus.cc:357] T 3d662e0ce13f447595e5f92b94de3018 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.262033 1173 raft_consensus.cc:383] T 3d662e0ce13f447595e5f92b94de3018 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.262099 1173 raft_consensus.cc:738] T 3d662e0ce13f447595e5f92b94de3018 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.262353 1173 consensus_queue.cc:260] T 3d662e0ce13f447595e5f92b94de3018 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.262879 1173 ts_tablet_manager.cc:1428] T 3d662e0ce13f447595e5f92b94de3018 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:43.263116 1173 tablet_bootstrap.cc:492] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:43.263891 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 781a9943f705477cb6eae16df4468d2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:43.264148 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a88463991294bd8a335337f8b072c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:43.264300 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e7a2e88c374e949ced92c557225cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:43.264485 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e7a2e88c374e949ced92c557225cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:43.264567 1173 tablet_bootstrap.cc:654] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.265560 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e7a2e88c374e949ced92c557225cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:43.245536 448 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40610 (request call id 189) took 2119 ms. Trace:
I20250112 02:52:43.269591 448 rpcz_store.cc:276] 0112 02:52:41.125766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.710693 (+584927us) service_pool.cc:225] Handling call
0112 02:52:41.711184 (+ 491us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:41.711343 (+ 159us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:52:41.711432 (+ 89us) catalog_manager.cc:2762] Locking tablets
0112 02:52:41.766208 (+ 54776us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:52:41.803407 (+ 37199us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:52:41.956923 (+153516us) catalog_manager.cc:2810] Committing in-memory state
0112 02:52:43.245046 (+1288123us) catalog_manager.cc:2814] Aborting table tasks
0112 02:52:43.245494 (+ 448us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:41.782148 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.782258 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.782264 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:41.782267 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:41.782410 (+ 143us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.782420 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.782426 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:41.782522 (+ 96us) tablet.cc:698] Row locks acquired
0112 02:52:41.782524 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:41.782557 (+ 33us) write_op.cc:270] Start()
0112 02:52:41.782586 (+ 29us) write_op.cc:276] Timestamp: P: 1736650361782507 usec, L: 0
0112 02:52:41.782590 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:41.782741 (+ 151us) log.cc:844] Serialized 725 byte log entry
0112 02:52:41.801625 (+ 18884us) op_driver.cc:464] REPLICATION: finished
0112 02:52:41.801712 (+ 87us) write_op.cc:301] APPLY: starting
0112 02:52:41.801723 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:41.801752 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:41.801770 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:41.801814 (+ 44us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:41.801836 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:41.801849 (+ 13us) write_op.cc:312] APPLY: finished
0112 02:52:41.801894 (+ 45us) log.cc:844] Serialized 29 byte log entry
0112 02:52:41.801935 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.801948 (+ 13us) write_op.cc:454] Released schema lock
0112 02:52:41.801953 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:41.802004 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.802007 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":45,"apply.run_cpu_time_us":285,"apply.run_wall_time_us":284,"num_ops":2,"prepare.queue_time_us":65,"prepare.run_cpu_time_us":708,"prepare.run_wall_time_us":708,"replication_time_us":19012}]]}
I20250112 02:52:43.270525 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_f7fd6e154c104f7ab66a01e7f8202da6" } modify_external_catalogs: true
I20250112 02:52:43.270725 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_f7fd6e154c104f7ab66a01e7f8202da6" } modify_external_catalogs: true
I20250112 02:52:43.271550 1181 consensus_queue.cc:260] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.245618 454 rpcz_store.cc:276] 0112 02:52:41.246452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.930036 (+683584us) service_pool.cc:225] Handling call
0112 02:52:41.930451 (+ 415us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:41.930606 (+ 155us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:52:41.930705 (+ 99us) catalog_manager.cc:2762] Locking tablets
0112 02:52:41.930835 (+ 130us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:52:41.961287 (+ 30452us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:52:41.961380 (+ 93us) catalog_manager.cc:2810] Committing in-memory state
0112 02:52:43.245013 (+1283633us) catalog_manager.cc:2814] Aborting table tasks
0112 02:52:43.245476 (+ 463us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:41.931526 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:41.931621 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:41.931627 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:41.931630 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:41.931717 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:41.931725 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:41.931729 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:41.931804 (+ 75us) tablet.cc:698] Row locks acquired
0112 02:52:41.931806 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:41.931832 (+ 26us) write_op.cc:270] Start()
0112 02:52:41.931856 (+ 24us) write_op.cc:276] Timestamp: P: 1736650361931755 usec, L: 0
0112 02:52:41.931860 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:41.931989 (+ 129us) log.cc:844] Serialized 759 byte log entry
0112 02:52:41.937556 (+ 5567us) op_driver.cc:464] REPLICATION: finished
0112 02:52:41.937631 (+ 75us) write_op.cc:301] APPLY: starting
0112 02:52:41.937641 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:41.937673 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:41.937698 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:41.937744 (+ 46us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:41.937768 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:41.937774 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:41.937818 (+ 44us) log.cc:844] Serialized 29 byte log entry
0112 02:52:41.937848 (+ 30us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.937860 (+ 12us) write_op.cc:454] Released schema lock
0112 02:52:41.937864 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:41.937950 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:41.937953 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":41,"apply.run_cpu_time_us":313,"apply.run_wall_time_us":313,"num_ops":2,"prepare.queue_time_us":176,"prepare.run_cpu_time_us":519,"prepare.run_wall_time_us":518,"replication_time_us":5675}]]}
I20250112 02:52:43.271944 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 2361bc6bd8164c2daa37be6b0845f4bd from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:43.272054 1181 ts_tablet_manager.cc:1428] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250112 02:52:43.273497 2876 raft_consensus.cc:491] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.273645 2876 raft_consensus.cc:513] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.274143 2876 leader_election.cc:290] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.274657 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61eebc5f22664712af6e74e82809ca87" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:43.275063 855 leader_election.cc:343] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.275408 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61eebc5f22664712af6e74e82809ca87" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:43.275799 854 leader_election.cc:343] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.275918 854 leader_election.cc:304] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:43.275954 1181 tablet_bootstrap.cc:492] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:43.276255 2876 raft_consensus.cc:2743] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.279376 1181 tablet_bootstrap.cc:654] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.279610 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a3af0ea4e954053ae89814f3e74e6b7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:43.280133 854 leader_election.cc:343] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.280247 854 leader_election.cc:304] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:43.280576 2876 raft_consensus.cc:2743] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.281351 2876 raft_consensus.cc:491] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.281487 2876 raft_consensus.cc:513] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.281991 2876 leader_election.cc:290] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.282459 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecf730b6985404a8134b497a4b792c8" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:43.282922 855 leader_election.cc:343] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.283800 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecf730b6985404a8134b497a4b792c8" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:43.284260 854 leader_election.cc:343] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.284370 854 leader_election.cc:304] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:43.286739 2876 raft_consensus.cc:2743] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.288053 1177 tablet_bootstrap.cc:654] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.299167 2024 ts_tablet_manager.cc:1918] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:43.299284 2024 log.cc:1198] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/8a88463991294bd8a335337f8b072c3d
I20250112 02:52:43.299691 2024 ts_tablet_manager.cc:1939] T 8a88463991294bd8a335337f8b072c3d P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:43.300843 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 8a88463991294bd8a335337f8b072c3d (table default.Table_56ea07abda424090a7942efabc871092 [id=44f7673679d14145a8cf65abd82094ce]) successfully deleted
I20250112 02:52:43.315641 2965 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 436220c23fa948efb5b6ca15875d8b0e)
I20250112 02:52:43.317471 2965 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } }
I20250112 02:52:43.318167 2965 leader_election.cc:290] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 436220c23fa948efb5b6ca15875d8b0e (127.0.104.62:42341), dd25f03f25df49178a5c53c636ecf304 (127.0.104.60:39987)
W20250112 02:52:43.318190 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 781a9943f705477cb6eae16df4468d2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 781a9943f705477cb6eae16df4468d2f already in progress: creating tablet
I20250112 02:52:43.319679 2024 tablet_replica.cc:331] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:43.319911 2024 raft_consensus.cc:2238] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:43.320137 2024 raft_consensus.cc:2267] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:43.320703 2024 ts_tablet_manager.cc:1905] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:43.320627 1393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4a49a5756b143fcb767c3f41463f8d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 704 } } ignore_live_leader: false dest_uuid: "dd25f03f25df49178a5c53c636ecf304" is_pre_election: true
I20250112 02:52:43.325227 1177 tablet_bootstrap.cc:492] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:43.326232 464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f4a49a5756b143fcb767c3f41463f8d9" candidate_term: 5 candidate_status { last_received { term: 4 index: 704 } } ignore_live_leader: false dest_uuid: "436220c23fa948efb5b6ca15875d8b0e" is_pre_election: true
I20250112 02:52:43.326820 2246 leader_election.cc:304] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4a49a5756b143fcb767c3f41463f8d9; no voters: 436220c23fa948efb5b6ca15875d8b0e, dd25f03f25df49178a5c53c636ecf304
I20250112 02:52:43.327098 2965 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250112 02:52:43.327224 1177 ts_tablet_manager.cc:1397] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.000s
I20250112 02:52:43.329373 1177 raft_consensus.cc:357] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.332942 2609 raft_consensus.cc:491] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.333071 2609 raft_consensus.cc:513] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.333638 2609 leader_election.cc:290] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:43.336163 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b81aee21510452881948e41d7aba9be" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:43.336330 910 raft_consensus.cc:2463] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:43.336756 989 leader_election.cc:304] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:43.336938 2960 tablet_replica.cc:331] T 781a9943f705477cb6eae16df4468d2f P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
W20250112 02:52:43.329864 714 consensus_peers.cc:487] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a88463991294bd8a335337f8b072c3d. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:43.337054 2745 raft_consensus.cc:2798] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:43.337137 2745 raft_consensus.cc:491] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:43.337213 2745 raft_consensus.cc:3054] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.334110 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b81aee21510452881948e41d7aba9be" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:43.337425 1177 raft_consensus.cc:383] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.337507 1177 raft_consensus.cc:738] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.337978 2960 raft_consensus.cc:2238] T 781a9943f705477cb6eae16df4468d2f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:43.337675 1177 consensus_queue.cc:260] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.338476 2960 raft_consensus.cc:2267] T 781a9943f705477cb6eae16df4468d2f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:43.338888 1177 ts_tablet_manager.cc:1428] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250112 02:52:43.339571 2673 raft_consensus.cc:491] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.339689 2673 raft_consensus.cc:513] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:43.339823 991 leader_election.cc:343] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.340178 2673 leader_election.cc:290] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.340279 2673 raft_consensus.cc:491] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.340363 2673 raft_consensus.cc:513] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.340801 2673 leader_election.cc:290] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.341090 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d662e0ce13f447595e5f92b94de3018" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:43.340415 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3771166b77e4f90aaa116ab1a9dc299" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:43.341253 776 raft_consensus.cc:2463] T 3d662e0ce13f447595e5f92b94de3018 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:43.341334 1177 tablet_bootstrap.cc:492] T 03a32b424ec4431d88f6af45dd09089f P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:43.341637 855 leader_election.cc:343] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.341873 855 leader_election.cc:304] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:43.342232 2876 raft_consensus.cc:2798] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:43.342316 2876 raft_consensus.cc:491] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:43.342394 2876 raft_consensus.cc:3054] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.342557 2960 ts_tablet_manager.cc:1905] T 781a9943f705477cb6eae16df4468d2f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:43.345566 2876 raft_consensus.cc:513] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.345973 1177 tablet_bootstrap.cc:654] T 03a32b424ec4431d88f6af45dd09089f P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.347285 2876 leader_election.cc:290] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.347110 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d662e0ce13f447595e5f92b94de3018" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:43.347807 776 raft_consensus.cc:3054] T 3d662e0ce13f447595e5f92b94de3018 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.347816 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d662e0ce13f447595e5f92b94de3018" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:43.347868 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3771166b77e4f90aaa116ab1a9dc299" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:43.348006 1049 raft_consensus.cc:2463] T 3d662e0ce13f447595e5f92b94de3018 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:43.348313 2745 raft_consensus.cc:513] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.348838 2745 leader_election.cc:290] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:43.349339 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b81aee21510452881948e41d7aba9be" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
W20250112 02:52:43.349728 991 leader_election.cc:343] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.350255 776 raft_consensus.cc:2463] T 3d662e0ce13f447595e5f92b94de3018 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:43.350275 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d662e0ce13f447595e5f92b94de3018" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:43.350451 1049 raft_consensus.cc:3054] T 3d662e0ce13f447595e5f92b94de3018 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:43.350435 854 leader_election.cc:343] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.350561 854 leader_election.cc:304] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:43.350662 855 leader_election.cc:304] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:43.350863 2673 raft_consensus.cc:2743] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.351050 2673 raft_consensus.cc:2798] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:43.351153 2673 raft_consensus.cc:695] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:43.351382 2673 consensus_queue.cc:237] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.352339 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b81aee21510452881948e41d7aba9be" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:43.352520 910 raft_consensus.cc:3054] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.353243 1049 raft_consensus.cc:2463] T 3d662e0ce13f447595e5f92b94de3018 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:43.355002 910 raft_consensus.cc:2463] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:43.355785 2849 catalog_manager.cc:1260] Loaded cluster ID: 273ff2f1d16946fcac5345b54ec7471c
I20250112 02:52:43.356454 989 leader_election.cc:304] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:43.356695 2326 raft_consensus.cc:2798] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:43.356818 2326 raft_consensus.cc:695] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:43.356879 2960 ts_tablet_manager.cc:1918] T 781a9943f705477cb6eae16df4468d2f P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:43.356976 2960 log.cc:1198] T 781a9943f705477cb6eae16df4468d2f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/781a9943f705477cb6eae16df4468d2f
I20250112 02:52:43.357038 2326 consensus_queue.cc:237] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.357249 2960 ts_tablet_manager.cc:1939] T 781a9943f705477cb6eae16df4468d2f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:43.358110 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 781a9943f705477cb6eae16df4468d2f (table default.Table_104ccd5f390f47a5878c67ea6de9d554 [id=872482df2ee94056a7d29ce227797f16]) successfully deleted
I20250112 02:52:43.358237 2849 catalog_manager.cc:1553] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: loading cluster ID for follower catalog manager: success
I20250112 02:52:43.359097 1177 tablet_bootstrap.cc:492] T 03a32b424ec4431d88f6af45dd09089f P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:43.359807 1177 ts_tablet_manager.cc:1397] T 03a32b424ec4431d88f6af45dd09089f P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20250112 02:52:43.363655 2960 tablet_replica.cc:331] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:43.364020 2960 raft_consensus.cc:2238] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:43.364292 2745 raft_consensus.cc:491] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.364421 2745 raft_consensus.cc:513] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.364516 2960 raft_consensus.cc:2267] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:43.364825 1177 raft_consensus.cc:357] T 03a32b424ec4431d88f6af45dd09089f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.364928 2745 leader_election.cc:290] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:43.364990 1177 raft_consensus.cc:383] T 03a32b424ec4431d88f6af45dd09089f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.365329 1177 raft_consensus.cc:738] T 03a32b424ec4431d88f6af45dd09089f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.365362 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a32b424ec4431d88f6af45dd09089f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:43.365785 2960 ts_tablet_manager.cc:1905] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:43.366000 1177 consensus_queue.cc:260] T 03a32b424ec4431d88f6af45dd09089f P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.366245 776 raft_consensus.cc:2463] T 03a32b424ec4431d88f6af45dd09089f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:43.366659 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a32b424ec4431d88f6af45dd09089f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:43.366886 917 raft_consensus.cc:2463] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:43.366817 1177 ts_tablet_manager.cc:1428] T 03a32b424ec4431d88f6af45dd09089f P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 02:52:43.367455 991 leader_election.cc:304] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:43.367534 1177 tablet_bootstrap.cc:492] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:43.368216 2609 raft_consensus.cc:2798] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:43.368340 2609 raft_consensus.cc:491] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:43.368469 2609 raft_consensus.cc:3054] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.370893 1177 tablet_bootstrap.cc:654] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.371026 2609 raft_consensus.cc:513] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.371872 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a32b424ec4431d88f6af45dd09089f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:43.371996 776 raft_consensus.cc:3054] T 03a32b424ec4431d88f6af45dd09089f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.373750 2609 leader_election.cc:290] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:43.374312 776 raft_consensus.cc:2463] T 03a32b424ec4431d88f6af45dd09089f P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:43.374728 991 leader_election.cc:304] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:43.375008 2745 raft_consensus.cc:2798] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:43.375140 2745 raft_consensus.cc:695] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:43.375450 2745 consensus_queue.cc:237] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.377255 1177 tablet_bootstrap.cc:492] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:43.377416 1177 ts_tablet_manager.cc:1397] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20250112 02:52:43.378012 1177 raft_consensus.cc:357] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.378230 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a32b424ec4431d88f6af45dd09089f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:43.378387 910 raft_consensus.cc:3054] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.380013 1177 raft_consensus.cc:383] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.380226 1177 raft_consensus.cc:738] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.380580 1177 consensus_queue.cc:260] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.381083 910 raft_consensus.cc:2463] T 03a32b424ec4431d88f6af45dd09089f P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:43.381271 1177 ts_tablet_manager.cc:1428] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250112 02:52:43.381536 2906 consensus_queue.cc:1035] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.381935 2960 ts_tablet_manager.cc:1918] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:43.382469 1177 tablet_bootstrap.cc:492] T 26577b614e32458e8bc5d6830c8c2e40 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:43.383915 2960 log.cc:1198] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/8a88463991294bd8a335337f8b072c3d
I20250112 02:52:43.384577 2960 ts_tablet_manager.cc:1939] T 8a88463991294bd8a335337f8b072c3d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:43.385784 1177 tablet_bootstrap.cc:654] T 26577b614e32458e8bc5d6830c8c2e40 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.386934 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 8a88463991294bd8a335337f8b072c3d (table default.Table_56ea07abda424090a7942efabc871092 [id=44f7673679d14145a8cf65abd82094ce]) successfully deleted
I20250112 02:52:43.387233 2179 consensus_queue.cc:1035] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.388706 2960 tablet_replica.cc:331] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:43.389070 2960 raft_consensus.cc:2238] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:43.389796 2960 raft_consensus.cc:2267] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:43.390802 2960 ts_tablet_manager.cc:1905] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:43.393520 1177 tablet_bootstrap.cc:492] T 26577b614e32458e8bc5d6830c8c2e40 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:43.395347 1177 ts_tablet_manager.cc:1397] T 26577b614e32458e8bc5d6830c8c2e40 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.001s
I20250112 02:52:43.396088 1177 raft_consensus.cc:357] T 26577b614e32458e8bc5d6830c8c2e40 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.396967 1177 raft_consensus.cc:383] T 26577b614e32458e8bc5d6830c8c2e40 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.397192 1177 raft_consensus.cc:738] T 26577b614e32458e8bc5d6830c8c2e40 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.397518 2876 raft_consensus.cc:491] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.397691 2876 raft_consensus.cc:513] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.398332 2876 leader_election.cc:290] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.398965 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95f98ae7bdf42518bbff55e257fea88" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:43.399489 854 leader_election.cc:343] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.399921 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95f98ae7bdf42518bbff55e257fea88" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:43.399981 1177 consensus_queue.cc:260] T 26577b614e32458e8bc5d6830c8c2e40 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.400514 1177 ts_tablet_manager.cc:1428] T 26577b614e32458e8bc5d6830c8c2e40 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.005s user 0.000s sys 0.000s
W20250112 02:52:43.400516 855 leader_election.cc:343] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.400640 855 leader_election.cc:304] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:43.400810 1177 tablet_bootstrap.cc:492] T ea18666ec324403b9f10a099688438b8 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:43.403350 1177 tablet_bootstrap.cc:654] T ea18666ec324403b9f10a099688438b8 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.403816 2876 raft_consensus.cc:2743] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.407932 2960 ts_tablet_manager.cc:1918] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:43.411559 2609 raft_consensus.cc:491] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.411746 2609 raft_consensus.cc:513] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.412575 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709517ae520640208abe85fa14a5301e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:43.413033 991 leader_election.cc:343] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.413865 1177 tablet_bootstrap.cc:492] T ea18666ec324403b9f10a099688438b8 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:43.414134 2609 leader_election.cc:290] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:43.414402 1177 ts_tablet_manager.cc:1397] T ea18666ec324403b9f10a099688438b8 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.002s
I20250112 02:52:43.415086 1177 raft_consensus.cc:357] T ea18666ec324403b9f10a099688438b8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.415939 1177 raft_consensus.cc:383] T ea18666ec324403b9f10a099688438b8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.416237 1177 raft_consensus.cc:738] T ea18666ec324403b9f10a099688438b8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.416607 1177 consensus_queue.cc:260] T ea18666ec324403b9f10a099688438b8 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.414597 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709517ae520640208abe85fa14a5301e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:43.417596 2960 log.cc:1198] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/23e7a2e88c374e949ced92c557225cbf
I20250112 02:52:43.417747 917 raft_consensus.cc:2463] T 709517ae520640208abe85fa14a5301e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:43.418141 2960 ts_tablet_manager.cc:1939] T 23e7a2e88c374e949ced92c557225cbf P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:43.418220 989 leader_election.cc:304] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:43.418506 1177 ts_tablet_manager.cc:1428] T ea18666ec324403b9f10a099688438b8 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 02:52:43.418596 2609 raft_consensus.cc:2798] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:43.418715 2609 raft_consensus.cc:491] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:43.418829 2609 raft_consensus.cc:3054] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.418977 1177 tablet_bootstrap.cc:492] T 41d5799b5b36487da42dc43b1c70ae26 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:43.419968 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 23e7a2e88c374e949ced92c557225cbf (table default.Table_cd1f185d82b3474083246ec3702851fc [id=bfa9a504a8d34c86929efb0d15d049d0]) successfully deleted
I20250112 02:52:43.421176 2609 raft_consensus.cc:513] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.421464 2179 raft_consensus.cc:491] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.421604 2179 raft_consensus.cc:513] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.421980 1177 tablet_bootstrap.cc:654] T 41d5799b5b36487da42dc43b1c70ae26 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.422334 2179 leader_election.cc:290] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:43.423328 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f3be6bcbe8a45f0a8076b4842d90dc2" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:43.424439 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f3be6bcbe8a45f0a8076b4842d90dc2" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:43.425424 714 leader_election.cc:304] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:43.425770 2179 raft_consensus.cc:2743] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.436955 1177 tablet_bootstrap.cc:492] T 41d5799b5b36487da42dc43b1c70ae26 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:43.437122 1177 ts_tablet_manager.cc:1397] T 41d5799b5b36487da42dc43b1c70ae26 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.003s
I20250112 02:52:43.437868 1177 raft_consensus.cc:357] T 41d5799b5b36487da42dc43b1c70ae26 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.438095 1177 raft_consensus.cc:383] T 41d5799b5b36487da42dc43b1c70ae26 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.439182 2876 raft_consensus.cc:491] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.439321 2876 raft_consensus.cc:513] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.438151 1177 raft_consensus.cc:738] T 41d5799b5b36487da42dc43b1c70ae26 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.439896 2876 leader_election.cc:290] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:43.440062 1177 consensus_queue.cc:260] T 41d5799b5b36487da42dc43b1c70ae26 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.440335 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05000c6a48e0450d8e6685ca864f67f0" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:43.440814 855 leader_election.cc:343] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.440958 1177 ts_tablet_manager.cc:1428] T 41d5799b5b36487da42dc43b1c70ae26 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20250112 02:52:43.441462 854 leader_election.cc:343] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 05000c6a48e0450d8e6685ca864f67f0
I20250112 02:52:43.441499 1177 tablet_bootstrap.cc:492] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:43.441593 854 leader_election.cc:304] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:43.441820 2876 raft_consensus.cc:2743] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.445684 1177 tablet_bootstrap.cc:654] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.454304 2849 catalog_manager.cc:1575] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: acquiring CA information for follower catalog manager: success
I20250112 02:52:43.455657 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709517ae520640208abe85fa14a5301e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
W20250112 02:52:43.456207 991 leader_election.cc:343] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.456987 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709517ae520640208abe85fa14a5301e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:43.457177 917 raft_consensus.cc:3054] T 709517ae520640208abe85fa14a5301e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.459704 917 raft_consensus.cc:2463] T 709517ae520640208abe85fa14a5301e P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:43.460167 989 leader_election.cc:304] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:43.460426 2977 raft_consensus.cc:2798] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:43.460542 2977 raft_consensus.cc:695] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:43.460758 2977 consensus_queue.cc:237] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.462276 1177 tablet_bootstrap.cc:492] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:43.462744 1177 ts_tablet_manager.cc:1397] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20250112 02:52:43.464077 1177 raft_consensus.cc:357] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.464485 1177 raft_consensus.cc:383] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.464650 1177 raft_consensus.cc:738] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.465019 1177 consensus_queue.cc:260] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.465536 2978 consensus_queue.cc:1035] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.474241 1177 ts_tablet_manager.cc:1428] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20250112 02:52:43.474822 1177 tablet_bootstrap.cc:492] T 82b351c563d749df806ab5d0331a4ef2 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:43.476984 2609 leader_election.cc:290] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:43.477936 1177 tablet_bootstrap.cc:654] T 82b351c563d749df806ab5d0331a4ef2 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.485126 2978 consensus_queue.cc:1035] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.487933 1177 tablet_bootstrap.cc:492] T 82b351c563d749df806ab5d0331a4ef2 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:43.488071 1177 ts_tablet_manager.cc:1397] T 82b351c563d749df806ab5d0331a4ef2 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20250112 02:52:43.489771 1177 raft_consensus.cc:357] T 82b351c563d749df806ab5d0331a4ef2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.490527 1177 raft_consensus.cc:383] T 82b351c563d749df806ab5d0331a4ef2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.490830 1177 raft_consensus.cc:738] T 82b351c563d749df806ab5d0331a4ef2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.491210 1177 consensus_queue.cc:260] T 82b351c563d749df806ab5d0331a4ef2 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.491950 1177 ts_tablet_manager.cc:1428] T 82b351c563d749df806ab5d0331a4ef2 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:43.492622 1177 tablet_bootstrap.cc:492] T 61eac6b572ff47c185768bee7490edf7 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:43.494372 1177 tablet_bootstrap.cc:654] T 61eac6b572ff47c185768bee7490edf7 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.495669 774 raft_consensus.cc:3054] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.499738 2906 consensus_queue.cc:1035] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.507795 2959 tablet_replica.cc:331] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:43.507828 2876 raft_consensus.cc:491] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.507997 2959 raft_consensus.cc:2238] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:43.507953 2876 raft_consensus.cc:513] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.508270 2959 raft_consensus.cc:2267] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:43.508564 2876 leader_election.cc:290] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.508750 2959 ts_tablet_manager.cc:1905] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:43.510442 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a470f84d50df4cf0b4a1e5d248b4c32a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:43.510933 855 leader_election.cc:343] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.513470 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a470f84d50df4cf0b4a1e5d248b4c32a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:43.514004 854 leader_election.cc:343] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.514119 854 leader_election.cc:304] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:43.514499 2876 raft_consensus.cc:2743] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.522073 1050 raft_consensus.cc:3054] T 9ded8bcdda914ea6bc4f9bb92d413dee P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.522310 2865 tablet.cc:1721] T e4d42f1611de49d6b6e5aaa9be907054 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:43.526314 2853 tablet.cc:1721] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:43.527628 2179 consensus_queue.cc:1035] T 9ded8bcdda914ea6bc4f9bb92d413dee P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.537706 2024 ts_tablet_manager.cc:1918] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:43.537828 2024 log.cc:1198] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/23e7a2e88c374e949ced92c557225cbf
I20250112 02:52:43.538205 2024 ts_tablet_manager.cc:1939] T 23e7a2e88c374e949ced92c557225cbf P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:43.539196 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 23e7a2e88c374e949ced92c557225cbf (table default.Table_cd1f185d82b3474083246ec3702851fc [id=bfa9a504a8d34c86929efb0d15d049d0]) successfully deleted
I20250112 02:52:43.545645 1173 tablet_bootstrap.cc:492] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:43.545780 1173 ts_tablet_manager.cc:1397] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.283s user 0.020s sys 0.019s
I20250112 02:52:43.546568 2673 raft_consensus.cc:491] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.546697 2673 raft_consensus.cc:513] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.552042 2673 leader_election.cc:290] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.552417 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5450777104da48f1821575658fd31afd" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:43.553040 854 leader_election.cc:343] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.553195 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5450777104da48f1821575658fd31afd" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:43.553426 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 56ddc025a8314f6f84d229191eb56692 (table default.Table_d22db1cc1d354af495d9888de1bfc22a [id=2ddd5dd296b94bc8b96a97b66e775836], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 56ddc025a8314f6f84d229191eb56692 consensus state has no leader
W20250112 02:52:43.553761 855 leader_election.cc:343] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.553908 855 leader_election.cc:304] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:43.554440 2876 raft_consensus.cc:2743] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.561022 1173 raft_consensus.cc:357] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.561688 1173 raft_consensus.cc:383] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.561908 1173 raft_consensus.cc:738] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.562268 1173 consensus_queue.cc:260] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.563119 1173 ts_tablet_manager.cc:1428] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250112 02:52:43.563143 2745 consensus_queue.cc:1035] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.527711 1177 tablet_bootstrap.cc:492] T 61eac6b572ff47c185768bee7490edf7 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:43.565357 2228 consensus_queue.cc:1035] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.566903 1177 ts_tablet_manager.cc:1397] T 61eac6b572ff47c185768bee7490edf7 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.000s
I20250112 02:52:43.567445 1177 raft_consensus.cc:357] T 61eac6b572ff47c185768bee7490edf7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.567623 1177 raft_consensus.cc:383] T 61eac6b572ff47c185768bee7490edf7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.567669 1177 raft_consensus.cc:738] T 61eac6b572ff47c185768bee7490edf7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.569195 1177 consensus_queue.cc:260] T 61eac6b572ff47c185768bee7490edf7 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.569645 1177 ts_tablet_manager.cc:1428] T 61eac6b572ff47c185768bee7490edf7 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 02:52:43.569850 1177 tablet_bootstrap.cc:492] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:43.569988 2179 raft_consensus.cc:491] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.570091 2179 raft_consensus.cc:513] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.570617 2179 leader_election.cc:290] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:43.571352 1177 tablet_bootstrap.cc:654] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.572003 1173 tablet_bootstrap.cc:492] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:43.572790 2908 tablet.cc:1721] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:43.579201 1173 tablet_bootstrap.cc:654] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.581688 2673 raft_consensus.cc:491] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.581864 2673 raft_consensus.cc:513] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.582643 2673 leader_election.cc:290] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.584074 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26577b614e32458e8bc5d6830c8c2e40" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:43.584537 854 leader_election.cc:343] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:43.571842 714 leader_election.cc:343] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: bd5b3c712c0f448ea9b14ab93dd731ab
I20250112 02:52:43.583802 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26577b614e32458e8bc5d6830c8c2e40" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:43.605638 772 raft_consensus.cc:2463] T 26577b614e32458e8bc5d6830c8c2e40 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
W20250112 02:52:43.605794 713 leader_election.cc:343] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: bd5b3c712c0f448ea9b14ab93dd731ab
I20250112 02:52:43.606106 713 leader_election.cc:304] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:43.606226 2849 catalog_manager.cc:1603] T 00000000000000000000000000000000 P f4a49a5756b143fcb767c3f41463f8d9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 02:52:43.606881 2445 raft_consensus.cc:2743] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.607443 855 leader_election.cc:304] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:43.607684 2673 raft_consensus.cc:2798] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:43.607769 2673 raft_consensus.cc:491] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:43.607862 2673 raft_consensus.cc:3054] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.610356 2673 raft_consensus.cc:513] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.615759 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26577b614e32458e8bc5d6830c8c2e40" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
W20250112 02:52:43.616393 854 leader_election.cc:343] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.616575 2673 leader_election.cc:290] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.617525 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26577b614e32458e8bc5d6830c8c2e40" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:43.617674 772 raft_consensus.cc:3054] T 26577b614e32458e8bc5d6830c8c2e40 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.620095 772 raft_consensus.cc:2463] T 26577b614e32458e8bc5d6830c8c2e40 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:43.620504 855 leader_election.cc:304] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:43.621052 2876 raft_consensus.cc:2798] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:43.621215 2876 raft_consensus.cc:695] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:43.621449 2876 consensus_queue.cc:237] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.626809 2787 consensus_queue.cc:1035] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.630987 2959 ts_tablet_manager.cc:1918] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:43.631103 2959 log.cc:1198] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/8a88463991294bd8a335337f8b072c3d
I20250112 02:52:43.631482 2959 ts_tablet_manager.cc:1939] T 8a88463991294bd8a335337f8b072c3d P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:43.633931 2673 consensus_queue.cc:1035] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.634258 2179 raft_consensus.cc:491] T ea18666ec324403b9f10a099688438b8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.634388 2179 raft_consensus.cc:513] T ea18666ec324403b9f10a099688438b8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.634984 2179 leader_election.cc:290] T ea18666ec324403b9f10a099688438b8 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:43.635229 1173 tablet_bootstrap.cc:492] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:43.635380 1173 ts_tablet_manager.cc:1397] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.002s
I20250112 02:52:43.635857 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea18666ec324403b9f10a099688438b8" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:43.635890 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea18666ec324403b9f10a099688438b8" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:43.636353 910 raft_consensus.cc:2388] T ea18666ec324403b9f10a099688438b8 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 289afc116aa74408aa5bff74e11c50d8 in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
I20250112 02:52:43.637327 714 leader_election.cc:304] T ea18666ec324403b9f10a099688438b8 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:43.638140 2179 raft_consensus.cc:3054] T ea18666ec324403b9f10a099688438b8 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.638036 1173 raft_consensus.cc:357] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.638231 1173 raft_consensus.cc:383] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.638306 1173 raft_consensus.cc:738] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.639781 1177 tablet_bootstrap.cc:492] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:43.639882 1177 ts_tablet_manager.cc:1397] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.001s
I20250112 02:52:43.640456 1177 raft_consensus.cc:357] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.640595 2179 raft_consensus.cc:2743] T ea18666ec324403b9f10a099688438b8 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.640622 1177 raft_consensus.cc:383] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.640666 1177 raft_consensus.cc:738] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.641395 1173 consensus_queue.cc:260] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.642863 1173 ts_tablet_manager.cc:1428] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250112 02:52:43.643162 1173 tablet_bootstrap.cc:492] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:43.644737 1177 consensus_queue.cc:260] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.645874 1177 ts_tablet_manager.cc:1428] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250112 02:52:43.660636 826 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:43.669404 1173 tablet_bootstrap.cc:654] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.675019 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 8a88463991294bd8a335337f8b072c3d (table default.Table_56ea07abda424090a7942efabc871092 [id=44f7673679d14145a8cf65abd82094ce]) successfully deleted
W20250112 02:52:43.675873 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 781a9943f705477cb6eae16df4468d2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 781a9943f705477cb6eae16df4468d2f already in progress: creating tablet
I20250112 02:52:43.676132 2959 tablet_replica.cc:331] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:43.677067 2959 raft_consensus.cc:2238] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:43.680778 2959 raft_consensus.cc:2267] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:43.690793 2959 ts_tablet_manager.cc:1905] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:43.708068 1177 tablet_bootstrap.cc:492] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:43.713001 2959 ts_tablet_manager.cc:1918] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:43.739446 1173 tablet_bootstrap.cc:492] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:43.739604 2959 log.cc:1198] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/23e7a2e88c374e949ced92c557225cbf
I20250112 02:52:43.739603 1173 ts_tablet_manager.cc:1397] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.097s user 0.004s sys 0.001s
I20250112 02:52:43.739993 2959 ts_tablet_manager.cc:1939] T 23e7a2e88c374e949ced92c557225cbf P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:43.740269 1173 raft_consensus.cc:357] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.740439 1173 raft_consensus.cc:383] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.740509 1173 raft_consensus.cc:738] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.740721 1173 consensus_queue.cc:260] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.726792 1177 tablet_bootstrap.cc:654] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.741823 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 23e7a2e88c374e949ced92c557225cbf (table default.Table_cd1f185d82b3474083246ec3702851fc [id=bfa9a504a8d34c86929efb0d15d049d0]) successfully deleted
I20250112 02:52:43.742385 1173 ts_tablet_manager.cc:1428] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250112 02:52:43.742717 1173 tablet_bootstrap.cc:492] T 1a7717cdcb8a4f47b11ba7af54b57e3c P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:43.744107 1181 tablet_bootstrap.cc:492] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:43.744306 1181 ts_tablet_manager.cc:1397] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.469s user 0.033s sys 0.134s
I20250112 02:52:43.745198 1181 raft_consensus.cc:357] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.745412 1181 raft_consensus.cc:383] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.745571 1181 raft_consensus.cc:738] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.753261 1173 tablet_bootstrap.cc:654] T 1a7717cdcb8a4f47b11ba7af54b57e3c P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.760736 1177 tablet_bootstrap.cc:492] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:43.760877 1177 ts_tablet_manager.cc:1397] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.003s
I20250112 02:52:43.745888 1181 consensus_queue.cc:260] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.761445 1181 ts_tablet_manager.cc:1428] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250112 02:52:43.761421 1177 raft_consensus.cc:357] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.761615 1177 raft_consensus.cc:383] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.761689 1177 raft_consensus.cc:738] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.761888 1177 consensus_queue.cc:260] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.762022 1181 tablet_bootstrap.cc:492] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:43.763947 1181 tablet_bootstrap.cc:654] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.768358 1177 ts_tablet_manager.cc:1428] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20250112 02:52:43.768666 1177 tablet_bootstrap.cc:492] T 709517ae520640208abe85fa14a5301e P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:43.770390 1177 tablet_bootstrap.cc:654] T 709517ae520640208abe85fa14a5301e P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.772046 2985 raft_consensus.cc:491] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.772168 2985 raft_consensus.cc:513] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.772743 2985 leader_election.cc:290] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:43.773384 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f776e2a6ec704cf9b64483bc7d5b1dff" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:43.773561 776 raft_consensus.cc:2463] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:43.773926 991 leader_election.cc:304] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:43.781878 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:43.783983 447 catalog_manager.cc:5526] T 2a6650515ea540a8bbf21e2f88363360 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:43.784870 965 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:43.785843 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:43.786972 447 catalog_manager.cc:5526] T 7f3be6bcbe8a45f0a8076b4842d90dc2 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:43.788638 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:43.789578 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet e4d42f1611de49d6b6e5aaa9be907054 (table default.Table_f952a0479e8d478bbd9484fc5bf041c8 [id=c01b653cd9fe45b2af721eb97936c5d4]). Expected version 1 got 0
I20250112 02:52:43.790270 447 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:43.791013 447 catalog_manager.cc:5526] T 7ad9c47fcb534619949971796b60eba2 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:43.798702 776 raft_consensus.cc:3054] T 41d5799b5b36487da42dc43b1c70ae26 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.801909 2876 consensus_queue.cc:1035] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.809748 1173 tablet_bootstrap.cc:492] T 1a7717cdcb8a4f47b11ba7af54b57e3c P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:43.809892 1173 ts_tablet_manager.cc:1397] T 1a7717cdcb8a4f47b11ba7af54b57e3c P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.067s user 0.000s sys 0.003s
I20250112 02:52:43.810509 1173 raft_consensus.cc:357] T 1a7717cdcb8a4f47b11ba7af54b57e3c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.810732 1173 raft_consensus.cc:383] T 1a7717cdcb8a4f47b11ba7af54b57e3c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.810802 1173 raft_consensus.cc:738] T 1a7717cdcb8a4f47b11ba7af54b57e3c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.811010 1173 consensus_queue.cc:260] T 1a7717cdcb8a4f47b11ba7af54b57e3c P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.811491 1173 ts_tablet_manager.cc:1428] T 1a7717cdcb8a4f47b11ba7af54b57e3c P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:43.811729 1173 tablet_bootstrap.cc:492] T d63d411f906b4d4eb68343bdcc493e05 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:43.812966 1173 tablet_bootstrap.cc:654] T d63d411f906b4d4eb68343bdcc493e05 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.820156 453 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 196) took 2496 ms. Trace:
I20250112 02:52:43.820283 453 rpcz_store.cc:276] 0112 02:52:41.324073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:43.252590 (+1928517us) service_pool.cc:225] Handling call
0112 02:52:43.253590 (+ 1000us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:43.254094 (+ 504us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:43.819883 (+565789us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:43.820017 (+ 134us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:43.820114 (+ 97us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:43.724222 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:43.724323 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:43.724328 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:43.724330 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:43.724402 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:43.724411 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:43.724423 (+ 12us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:43.724484 (+ 61us) tablet.cc:698] Row locks acquired
0112 02:52:43.724486 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:43.724513 (+ 27us) write_op.cc:270] Start()
0112 02:52:43.724537 (+ 24us) write_op.cc:276] Timestamp: P: 1736650363724363 usec, L: 0
0112 02:52:43.724540 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:43.724685 (+ 145us) log.cc:844] Serialized 433 byte log entry
0112 02:52:43.818426 (+ 93741us) op_driver.cc:464] REPLICATION: finished
0112 02:52:43.819281 (+ 855us) write_op.cc:301] APPLY: starting
0112 02:52:43.819293 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:43.819325 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:43.819353 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:43.819436 (+ 83us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:43.819485 (+ 49us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:43.819492 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:43.819567 (+ 75us) log.cc:844] Serialized 29 byte log entry
0112 02:52:43.819779 (+ 212us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:43.819798 (+ 19us) write_op.cc:454] Released schema lock
0112 02:52:43.819806 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":815,"num_ops":2,"prepare.queue_time_us":469703,"prepare.run_cpu_time_us":508,"prepare.run_wall_time_us":506,"replication_time_us":93860}]]}
W20250112 02:52:43.821287 989 leader_election.cc:343] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: f776e2a6ec704cf9b64483bc7d5b1dff
W20250112 02:52:43.821633 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 47616.3us
I20250112 02:52:43.824000 2985 consensus_queue.cc:1035] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.824796 3000 consensus_queue.cc:1035] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.838569 1181 tablet_bootstrap.cc:492] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:43.838729 1181 ts_tablet_manager.cc:1397] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.077s user 0.008s sys 0.000s
I20250112 02:52:43.839524 1181 raft_consensus.cc:357] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.839763 1181 raft_consensus.cc:383] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.839834 1181 raft_consensus.cc:738] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.840016 1181 consensus_queue.cc:260] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.841563 1181 ts_tablet_manager.cc:1428] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:43.842244 1181 tablet_bootstrap.cc:492] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:43.844204 1181 tablet_bootstrap.cc:654] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.848294 1177 tablet_bootstrap.cc:492] T 709517ae520640208abe85fa14a5301e P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:43.848574 1177 ts_tablet_manager.cc:1397] T 709517ae520640208abe85fa14a5301e P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.080s user 0.002s sys 0.002s
I20250112 02:52:43.845278 3000 raft_consensus.cc:491] T 3d418cda9ed5449fa45690130cd6d5bc P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.848748 3000 raft_consensus.cc:513] T 3d418cda9ed5449fa45690130cd6d5bc P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.849253 1177 raft_consensus.cc:357] T 709517ae520640208abe85fa14a5301e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.849364 3000 leader_election.cc:290] T 3d418cda9ed5449fa45690130cd6d5bc P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.849442 1177 raft_consensus.cc:383] T 709517ae520640208abe85fa14a5301e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.849494 1177 raft_consensus.cc:738] T 709517ae520640208abe85fa14a5301e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.849692 1177 consensus_queue.cc:260] T 709517ae520640208abe85fa14a5301e P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.849882 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d418cda9ed5449fa45690130cd6d5bc" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:43.850703 855 leader_election.cc:343] T 3d418cda9ed5449fa45690130cd6d5bc P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.850747 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d418cda9ed5449fa45690130cd6d5bc" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:43.851154 1177 ts_tablet_manager.cc:1428] T 709517ae520640208abe85fa14a5301e P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250112 02:52:43.851217 854 leader_election.cc:343] T 3d418cda9ed5449fa45690130cd6d5bc P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.851322 854 leader_election.cc:304] T 3d418cda9ed5449fa45690130cd6d5bc P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:43.851449 1177 tablet_bootstrap.cc:492] T a3ffdc237f8640f0b378809f93fc6292 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:43.851733 3000 raft_consensus.cc:2743] T 3d418cda9ed5449fa45690130cd6d5bc P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.853022 1177 tablet_bootstrap.cc:654] T a3ffdc237f8640f0b378809f93fc6292 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.853466 2445 raft_consensus.cc:491] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.861934 2445 raft_consensus.cc:513] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.862493 2445 leader_election.cc:290] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.863061 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a3af0ea4e954053ae89814f3e74e6b7" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:43.863247 910 raft_consensus.cc:2463] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:43.866739 713 leader_election.cc:304] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:43.867157 2179 raft_consensus.cc:2798] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:43.867218 2179 raft_consensus.cc:491] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:43.867269 2179 raft_consensus.cc:3054] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.870998 2451 raft_consensus.cc:491] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.871127 2451 raft_consensus.cc:513] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.871767 2451 leader_election.cc:290] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:43.872486 714 leader_election.cc:343] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: b124c15736084e6d8f69b4fb7c41cf39
W20250112 02:52:43.872676 713 leader_election.cc:343] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: b124c15736084e6d8f69b4fb7c41cf39
I20250112 02:52:43.872759 713 leader_election.cc:304] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:43.873143 2451 raft_consensus.cc:2743] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.874158 2179 raft_consensus.cc:513] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.874706 2179 leader_election.cc:290] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.875174 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a3af0ea4e954053ae89814f3e74e6b7" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:43.875262 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a3af0ea4e954053ae89814f3e74e6b7" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
W20250112 02:52:43.875654 714 leader_election.cc:343] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.878371 1173 tablet_bootstrap.cc:492] T d63d411f906b4d4eb68343bdcc493e05 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:43.879174 910 raft_consensus.cc:3054] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.880684 1173 ts_tablet_manager.cc:1397] T d63d411f906b4d4eb68343bdcc493e05 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.000s
I20250112 02:52:43.881350 1173 raft_consensus.cc:357] T d63d411f906b4d4eb68343bdcc493e05 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.881505 1173 raft_consensus.cc:383] T d63d411f906b4d4eb68343bdcc493e05 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.881546 1173 raft_consensus.cc:738] T d63d411f906b4d4eb68343bdcc493e05 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.880172 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a3af0ea4e954053ae89814f3e74e6b7" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:43.881925 714 leader_election.cc:343] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.882581 910 raft_consensus.cc:2463] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:43.883040 713 leader_election.cc:304] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:43.883144 2978 raft_consensus.cc:2798] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:43.883226 2978 raft_consensus.cc:491] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:43.883283 2978 raft_consensus.cc:3054] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.883399 2445 raft_consensus.cc:2798] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:43.883533 2445 raft_consensus.cc:695] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:43.883744 2445 consensus_queue.cc:237] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.885599 2978 raft_consensus.cc:513] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.886374 2978 leader_election.cc:290] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:43.887526 2978 consensus_queue.cc:1035] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:52:43.888921 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 115010us
W20250112 02:52:43.888947 989 leader_election.cc:343] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: f776e2a6ec704cf9b64483bc7d5b1dff
I20250112 02:52:43.889338 1173 consensus_queue.cc:260] T d63d411f906b4d4eb68343bdcc493e05 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.892216 1173 ts_tablet_manager.cc:1428] T d63d411f906b4d4eb68343bdcc493e05 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250112 02:52:43.892776 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f776e2a6ec704cf9b64483bc7d5b1dff" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:43.892947 776 raft_consensus.cc:3054] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:43.893062 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3dab0466476c4a9aa265271c1abd91a4 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:43.893321 1173 tablet_bootstrap.cc:492] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:43.893566 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:43.894083 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_5d473cabff2e46e0b531fbce28d9d558 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 1 got 0
I20250112 02:52:43.894330 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:43.894706 2978 consensus_queue.cc:1035] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.894901 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:43.895706 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ca2165362ee541adac724b63f39b1489 (table default.Table_41b7056b6e9240928d0af52aec2e9338 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 2 got 0
I20250112 02:52:43.895938 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 56ddc025a8314f6f84d229191eb56692 (table default.Table_d22db1cc1d354af495d9888de1bfc22a [id=2ddd5dd296b94bc8b96a97b66e775836]). Expected version 1 got 0
I20250112 02:52:43.896117 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:43.896459 776 raft_consensus.cc:2463] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:43.896636 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 37d38f5e525440f8a0f7822620933435 (table default.Table_5aeb2d7bb68c46a589a48526dcc40aaf [id=9d267774bf59428e854651956fbd79be]). Expected version 1 got 0
I20250112 02:52:43.897106 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_a9907ed8bfe14625977ec98b67e9d599 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 2 got 0
I20250112 02:52:43.897450 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:43.898049 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:43.898730 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:43.898994 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:43.899658 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_b9dff6cb7bbd4d0a88de458399b0d2c0 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 2 got 0
I20250112 02:52:43.900300 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:43.900626 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]). Expected version 1 got 0
I20250112 02:52:43.901027 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5524d75599f04becbe6a94426adb7026 (table default.Table_e8cef38ebaa243ed83bfb41bcea1933f [id=b7739f28885a453d9ef3966511183c17]). Expected version 1 got 0
I20250112 02:52:43.901799 3000 raft_consensus.cc:491] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.901592 450 catalog_manager.cc:5526] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:43.901991 3000 raft_consensus.cc:513] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:43.902359 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:43.902791 3000 leader_election.cc:290] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:43.902943 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:43.903576 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet d95f98ae7bdf42518bbff55e257fea88 (table default.Table_2fcdbb05970f4a098f7ba3e287fa0348 [id=70c61753e9944e1fbd65e9ec41a48ee2]). Expected version 1 got 0
I20250112 02:52:43.903908 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]). Expected version 3 got 0
I20250112 02:52:43.903928 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7340a7a916c4b2fab32e0ba2d0ce322" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:43.904188 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:43.904726 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b9537fcd96eb41cc8f680075ec71137b (table default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29]). Expected version 3 got 0
I20250112 02:52:43.906083 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 17c54c5dc86446cf85974777c8145ef6 (table default.Table_8ab566cf9c8f45eba4dc9bb85264c823 [id=a622f7a973344a388b3275adbcce6234]). Expected version 1 got 0
I20250112 02:52:43.906458 450 catalog_manager.cc:5526] T 37862023095644bab2403576e8cb8c58 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:43.906978 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:43.907366 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet e4d42f1611de49d6b6e5aaa9be907054 (table default.Table_f952a0479e8d478bbd9484fc5bf041c8 [id=c01b653cd9fe45b2af721eb97936c5d4]). Expected version 1 got 0
W20250112 02:52:43.907955 854 leader_election.cc:343] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: b7340a7a916c4b2fab32e0ba2d0ce322
W20250112 02:52:43.908083 855 leader_election.cc:343] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.908242 855 leader_election.cc:304] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:43.908555 450 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:43.908605 3000 raft_consensus.cc:2743] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.909943 1181 tablet_bootstrap.cc:492] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:43.910072 1181 ts_tablet_manager.cc:1397] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.068s user 0.006s sys 0.000s
I20250112 02:52:43.911113 1181 raft_consensus.cc:357] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.911288 1181 raft_consensus.cc:383] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.911352 1181 raft_consensus.cc:738] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.911584 1181 consensus_queue.cc:260] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.913782 1181 ts_tablet_manager.cc:1428] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:43.922093 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 2361bc6bd8164c2daa37be6b0845f4bd
I20250112 02:52:43.922634 454 catalog_manager.cc:6596] ReplaceTablet: tablet 2361bc6bd8164c2daa37be6b0845f4bd deleted and replaced by tablet 154ecbf4175a42a7ad765780c0f902a5
I20250112 02:52:43.922740 454 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:40610 (request call id 198) took 2309 ms. Trace:
I20250112 02:52:43.922833 454 rpcz_store.cc:276] 0112 02:52:41.613086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:43.271882 (+1658796us) service_pool.cc:225] Handling call
0112 02:52:43.922711 (+650829us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:43.916974 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:43.917056 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:43.917061 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:43.917062 (+ 1us) tablet.cc:659] Decoding operations
0112 02:52:43.917147 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:43.917155 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:43.917159 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:43.917220 (+ 61us) tablet.cc:698] Row locks acquired
0112 02:52:43.917222 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:43.917245 (+ 23us) write_op.cc:270] Start()
0112 02:52:43.917266 (+ 21us) write_op.cc:276] Timestamp: P: 1736650363917195 usec, L: 0
0112 02:52:43.917269 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:43.917376 (+ 107us) log.cc:844] Serialized 656 byte log entry
0112 02:52:43.921038 (+ 3662us) op_driver.cc:464] REPLICATION: finished
0112 02:52:43.921108 (+ 70us) write_op.cc:301] APPLY: starting
0112 02:52:43.921117 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:43.921144 (+ 27us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:43.921167 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:43.921242 (+ 75us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:43.921279 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:43.921302 (+ 23us) write_op.cc:312] APPLY: finished
0112 02:52:43.921357 (+ 55us) log.cc:844] Serialized 29 byte log entry
0112 02:52:43.921398 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:43.921413 (+ 15us) write_op.cc:454] Released schema lock
0112 02:52:43.921429 (+ 16us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:43.921476 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:43.921488 (+ 12us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":39,"apply.run_cpu_time_us":362,"apply.run_wall_time_us":361,"num_ops":2,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":469,"prepare.run_wall_time_us":468,"raft.queue_time_us":45,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":166,"replication_time_us":3749}]]}
I20250112 02:52:43.924397 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 2361bc6bd8164c2daa37be6b0845f4bd with delete_type TABLET_DATA_DELETED (replaced by tablet 154ecbf4175a42a7ad765780c0f902a5) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:43.926198 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 2361bc6bd8164c2daa37be6b0845f4bd with delete_type TABLET_DATA_DELETED (replaced by tablet 154ecbf4175a42a7ad765780c0f902a5) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:43.926479 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 2361bc6bd8164c2daa37be6b0845f4bd with delete_type TABLET_DATA_DELETED (replaced by tablet 154ecbf4175a42a7ad765780c0f902a5) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:43.926766 2024 tablet_replica.cc:331] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:43.926932 2024 raft_consensus.cc:2238] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:43.927206 2024 raft_consensus.cc:2267] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:43.927719 2024 ts_tablet_manager.cc:1905] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:43.929929 2959 tablet_replica.cc:331] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:43.930100 2959 raft_consensus.cc:2238] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:43.930240 2959 raft_consensus.cc:2267] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:43.930711 2959 ts_tablet_manager.cc:1905] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:43.935611 3000 raft_consensus.cc:491] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:43.935753 3000 raft_consensus.cc:513] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.936435 3000 leader_election.cc:290] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:43.937052 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd81852962fb43449e35ef45dc1daaad" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:43.937600 854 leader_election.cc:343] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.939724 2024 ts_tablet_manager.cc:1918] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:43.939826 2024 log.cc:1198] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/2361bc6bd8164c2daa37be6b0845f4bd
I20250112 02:52:43.940145 2024 ts_tablet_manager.cc:1939] T 2361bc6bd8164c2daa37be6b0845f4bd P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:43.940160 1181 tablet_bootstrap.cc:492] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:43.940986 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 2361bc6bd8164c2daa37be6b0845f4bd (table default.Table_ad5f01f6a5274cc483c8f051059d36c3 [id=8be34ef1e0c54f7fa38ec57fb432b3db]) successfully deleted
I20250112 02:52:43.942178 1181 tablet_bootstrap.cc:654] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.942452 2959 ts_tablet_manager.cc:1918] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:43.942616 2959 log.cc:1198] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/2361bc6bd8164c2daa37be6b0845f4bd
I20250112 02:52:43.942986 2959 ts_tablet_manager.cc:1939] T 2361bc6bd8164c2daa37be6b0845f4bd P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:43.943949 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 2361bc6bd8164c2daa37be6b0845f4bd (table default.Table_ad5f01f6a5274cc483c8f051059d36c3 [id=8be34ef1e0c54f7fa38ec57fb432b3db]) successfully deleted
I20250112 02:52:43.943904 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd81852962fb43449e35ef45dc1daaad" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:43.944324 855 leader_election.cc:343] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:43.944416 855 leader_election.cc:304] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:43.944562 991 leader_election.cc:304] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:43.944751 2978 raft_consensus.cc:2798] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:43.944793 2673 raft_consensus.cc:2743] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:43.944849 2978 raft_consensus.cc:695] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:43.945034 2978 consensus_queue.cc:237] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.958846 2228 consensus_queue.cc:1035] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:43.963737 1181 tablet_bootstrap.cc:492] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:43.963899 1181 ts_tablet_manager.cc:1397] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.001s
I20250112 02:52:43.964672 1181 raft_consensus.cc:357] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.964874 1181 raft_consensus.cc:383] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.964926 1181 raft_consensus.cc:738] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.965131 1181 consensus_queue.cc:260] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.965791 1181 ts_tablet_manager.cc:1428] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 02:52:43.966044 1181 tablet_bootstrap.cc:492] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:43.968377 1177 tablet_bootstrap.cc:492] T a3ffdc237f8640f0b378809f93fc6292 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:43.968477 1177 ts_tablet_manager.cc:1397] T a3ffdc237f8640f0b378809f93fc6292 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.117s user 0.000s sys 0.008s
I20250112 02:52:43.970358 3028 tablet_replica.cc:331] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:43.971194 3028 raft_consensus.cc:2238] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:43.971959 3028 raft_consensus.cc:2267] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:43.973299 1177 raft_consensus.cc:357] T a3ffdc237f8640f0b378809f93fc6292 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.973965 3028 ts_tablet_manager.cc:1905] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:43.975307 1177 raft_consensus.cc:383] T a3ffdc237f8640f0b378809f93fc6292 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:43.979218 1177 raft_consensus.cc:738] T a3ffdc237f8640f0b378809f93fc6292 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:43.979668 1181 tablet_bootstrap.cc:654] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.979499 1177 consensus_queue.cc:260] T a3ffdc237f8640f0b378809f93fc6292 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:43.984948 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:43.985459 1177 ts_tablet_manager.cc:1428] T a3ffdc237f8640f0b378809f93fc6292 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250112 02:52:43.985733 1177 tablet_bootstrap.cc:492] T 1a7717cdcb8a4f47b11ba7af54b57e3c P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:43.990705 1177 tablet_bootstrap.cc:654] T 1a7717cdcb8a4f47b11ba7af54b57e3c P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:43.994273 3028 ts_tablet_manager.cc:1918] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:43.994385 3028 log.cc:1198] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/2361bc6bd8164c2daa37be6b0845f4bd
I20250112 02:52:43.994729 3028 ts_tablet_manager.cc:1939] T 2361bc6bd8164c2daa37be6b0845f4bd P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:43.997994 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_5d473cabff2e46e0b531fbce28d9d558 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 1 got 0
I20250112 02:52:43.998311 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:43.998867 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_a9907ed8bfe14625977ec98b67e9d599 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 2 got 0
I20250112 02:52:43.999289 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 2361bc6bd8164c2daa37be6b0845f4bd (table default.Table_ad5f01f6a5274cc483c8f051059d36c3 [id=8be34ef1e0c54f7fa38ec57fb432b3db]) successfully deleted
I20250112 02:52:44.000125 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_b9dff6cb7bbd4d0a88de458399b0d2c0 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 2 got 0
I20250112 02:52:44.001649 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 37d38f5e525440f8a0f7822620933435 (table default.Table_5aeb2d7bb68c46a589a48526dcc40aaf [id=9d267774bf59428e854651956fbd79be]). Expected version 1 got 0
I20250112 02:52:44.002308 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ca2165362ee541adac724b63f39b1489 (table default.Table_41b7056b6e9240928d0af52aec2e9338 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 2 got 0
I20250112 02:52:44.002352 3000 raft_consensus.cc:491] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.002476 3000 raft_consensus.cc:513] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.002964 1177 tablet_bootstrap.cc:492] T 1a7717cdcb8a4f47b11ba7af54b57e3c P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:44.003073 1177 ts_tablet_manager.cc:1397] T 1a7717cdcb8a4f47b11ba7af54b57e3c P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.003s
I20250112 02:52:44.003705 1173 tablet_bootstrap.cc:654] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.003655 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe57d374eb44c3fb1532484e214b654" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:44.003731 1177 raft_consensus.cc:357] T 1a7717cdcb8a4f47b11ba7af54b57e3c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.003911 1177 raft_consensus.cc:383] T 1a7717cdcb8a4f47b11ba7af54b57e3c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.003962 1177 raft_consensus.cc:738] T 1a7717cdcb8a4f47b11ba7af54b57e3c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
W20250112 02:52:44.004139 854 leader_election.cc:343] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.004158 1177 consensus_queue.cc:260] T 1a7717cdcb8a4f47b11ba7af54b57e3c P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.004376 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe57d374eb44c3fb1532484e214b654" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:44.004670 1177 ts_tablet_manager.cc:1428] T 1a7717cdcb8a4f47b11ba7af54b57e3c P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250112 02:52:44.004845 855 leader_election.cc:343] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.004933 855 leader_election.cc:304] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.005021 1177 tablet_bootstrap.cc:492] T 54215c55966a46e1ab96e352c4e3764e P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:44.005057 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:44.005146 2673 raft_consensus.cc:2743] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.005499 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:44.005851 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:44.006469 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:44.006632 1177 tablet_bootstrap.cc:654] T 54215c55966a46e1ab96e352c4e3764e P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.006714 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 56ddc025a8314f6f84d229191eb56692 (table default.Table_d22db1cc1d354af495d9888de1bfc22a [id=2ddd5dd296b94bc8b96a97b66e775836]). Expected version 1 got 0
I20250112 02:52:44.007016 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:44.007220 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3dab0466476c4a9aa265271c1abd91a4 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:44.007530 3000 leader_election.cc:290] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.007843 2978 raft_consensus.cc:491] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.007951 2978 raft_consensus.cc:513] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.008246 1181 tablet_bootstrap.cc:492] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:44.008368 1181 ts_tablet_manager.cc:1397] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.001s
I20250112 02:52:44.008493 2978 leader_election.cc:290] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:44.009181 1181 raft_consensus.cc:357] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.009379 1181 raft_consensus.cc:383] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 02:52:44.009352 989 leader_election.cc:343] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 82b351c563d749df806ab5d0331a4ef2
I20250112 02:52:44.009446 1181 raft_consensus.cc:738] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.009678 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b351c563d749df806ab5d0331a4ef2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:44.009639 1181 consensus_queue.cc:260] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.009837 772 raft_consensus.cc:2463] T 82b351c563d749df806ab5d0331a4ef2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:44.010113 1181 ts_tablet_manager.cc:1428] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:44.010330 1181 tablet_bootstrap.cc:492] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:44.010282 991 leader_election.cc:304] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:44.011849 1181 tablet_bootstrap.cc:654] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.013079 2978 raft_consensus.cc:2798] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:44.013152 2978 raft_consensus.cc:491] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:44.013229 2978 raft_consensus.cc:3054] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.014807 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:44.015632 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5524d75599f04becbe6a94426adb7026 (table default.Table_e8cef38ebaa243ed83bfb41bcea1933f [id=b7739f28885a453d9ef3966511183c17]). Expected version 1 got 0
I20250112 02:52:44.015901 2978 raft_consensus.cc:513] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.016469 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
W20250112 02:52:44.017186 989 leader_election.cc:343] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 82b351c563d749df806ab5d0331a4ef2
I20250112 02:52:44.017323 2978 leader_election.cc:290] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:44.017900 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b351c563d749df806ab5d0331a4ef2" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:44.017907 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:44.018040 772 raft_consensus.cc:3054] T 82b351c563d749df806ab5d0331a4ef2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.018680 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:44.019704 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:44.020601 772 raft_consensus.cc:2463] T 82b351c563d749df806ab5d0331a4ef2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:44.020931 991 leader_election.cc:304] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:44.021235 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet d95f98ae7bdf42518bbff55e257fea88 (table default.Table_2fcdbb05970f4a098f7ba3e287fa0348 [id=70c61753e9944e1fbd65e9ec41a48ee2]). Expected version 1 got 0
I20250112 02:52:44.021358 2978 raft_consensus.cc:2798] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:44.021551 2978 raft_consensus.cc:695] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:44.021756 2978 consensus_queue.cc:237] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.022389 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:44.023154 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:44.027171 453 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 17c54c5dc86446cf85974777c8145ef6 (table default.Table_8ab566cf9c8f45eba4dc9bb85264c823 [id=a622f7a973344a388b3275adbcce6234]). Expected version 1 got 0
I20250112 02:52:44.028328 1173 tablet_bootstrap.cc:492] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:44.028450 1173 ts_tablet_manager.cc:1397] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.135s user 0.002s sys 0.001s
I20250112 02:52:44.029062 1173 raft_consensus.cc:357] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.029227 1173 raft_consensus.cc:383] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.029284 1173 raft_consensus.cc:738] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.029476 1181 tablet_bootstrap.cc:492] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:44.029625 1181 ts_tablet_manager.cc:1397] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.009s
I20250112 02:52:44.030251 1181 raft_consensus.cc:357] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.030470 1181 raft_consensus.cc:383] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.030561 1181 raft_consensus.cc:738] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.030799 1181 consensus_queue.cc:260] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.031370 1181 ts_tablet_manager.cc:1428] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:44.031666 1181 tablet_bootstrap.cc:492] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:44.033886 1181 tablet_bootstrap.cc:654] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.034601 455 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 195) took 2726 ms. Trace:
I20250112 02:52:44.034684 455 rpcz_store.cc:276] 0112 02:52:41.308050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:43.250994 (+1942944us) service_pool.cc:225] Handling call
0112 02:52:43.251411 (+ 417us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:43.272291 (+ 20880us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:43.325636 (+ 53345us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:43.325747 (+ 111us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:43.825693 (+499946us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:44.034554 (+208861us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:43.724759 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:43.725801 (+ 1042us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:43.725807 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:43.725809 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:43.725876 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:43.725885 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:43.725891 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:43.725932 (+ 41us) tablet.cc:698] Row locks acquired
0112 02:52:43.725934 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:43.725958 (+ 24us) write_op.cc:270] Start()
0112 02:52:43.725978 (+ 20us) write_op.cc:276] Timestamp: P: 1736650363725810 usec, L: 0
0112 02:52:43.725980 (+ 2us) op_driver.cc:348] REPLICATION: starting
0112 02:52:43.727434 (+ 1454us) log.cc:844] Serialized 405 byte log entry
0112 02:52:43.818495 (+ 91061us) op_driver.cc:464] REPLICATION: finished
0112 02:52:43.825288 (+ 6793us) write_op.cc:301] APPLY: starting
0112 02:52:43.825302 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:43.825335 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:43.825376 (+ 41us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:43.825412 (+ 36us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:43.825444 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:43.825450 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:43.825530 (+ 80us) log.cc:844] Serialized 23 byte log entry
0112 02:52:43.825601 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:43.825614 (+ 13us) write_op.cc:454] Released schema lock
0112 02:52:43.825619 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:43.825691 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:43.825695 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":6703,"apply.run_cpu_time_us":401,"apply.run_wall_time_us":401,"num_ops":1,"prepare.queue_time_us":398577,"prepare.run_cpu_time_us":509,"prepare.run_wall_time_us":2758,"replication_time_us":92500}]]}
I20250112 02:52:44.038695 1173 consensus_queue.cc:260] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.045130 1177 tablet_bootstrap.cc:492] T 54215c55966a46e1ab96e352c4e3764e P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:44.045464 1177 ts_tablet_manager.cc:1397] T 54215c55966a46e1ab96e352c4e3764e P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.002s
I20250112 02:52:44.046159 1177 raft_consensus.cc:357] T 54215c55966a46e1ab96e352c4e3764e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.046345 1177 raft_consensus.cc:383] T 54215c55966a46e1ab96e352c4e3764e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.046397 1177 raft_consensus.cc:738] T 54215c55966a46e1ab96e352c4e3764e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.046608 1177 consensus_queue.cc:260] T 54215c55966a46e1ab96e352c4e3764e P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.047957 1173 ts_tablet_manager.cc:1428] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250112 02:52:44.048478 1177 ts_tablet_manager.cc:1428] T 54215c55966a46e1ab96e352c4e3764e P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 02:52:44.049921 1177 tablet_bootstrap.cc:492] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:44.057724 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 1a7717cdcb8a4f47b11ba7af54b57e3c
I20250112 02:52:44.058163 448 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40610 (request call id 197) took 2560 ms. Trace:
I20250112 02:52:44.058238 448 rpcz_store.cc:276] 0112 02:52:41.497835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:43.270348 (+1772513us) service_pool.cc:225] Handling call
0112 02:52:43.270836 (+ 488us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:43.270992 (+ 156us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:52:43.271052 (+ 60us) catalog_manager.cc:2762] Locking tablets
0112 02:52:43.775687 (+504635us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:52:43.856074 (+ 80387us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:52:43.856181 (+ 107us) catalog_manager.cc:2810] Committing in-memory state
0112 02:52:44.057682 (+201501us) catalog_manager.cc:2814] Aborting table tasks
0112 02:52:44.058120 (+ 438us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:43.782267 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:43.782344 (+ 77us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:43.782348 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:43.782350 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:43.782426 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:43.782433 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:43.782437 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:43.782524 (+ 87us) tablet.cc:698] Row locks acquired
0112 02:52:43.782526 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:43.782552 (+ 26us) write_op.cc:270] Start()
0112 02:52:43.782576 (+ 24us) write_op.cc:276] Timestamp: P: 1736650363782529 usec, L: 0
0112 02:52:43.782579 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:43.782702 (+ 123us) log.cc:844] Serialized 725 byte log entry
0112 02:52:43.854873 (+ 72171us) op_driver.cc:464] REPLICATION: finished
0112 02:52:43.855590 (+ 717us) write_op.cc:301] APPLY: starting
0112 02:52:43.855599 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:43.855629 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:43.855648 (+ 19us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:43.855765 (+ 117us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:43.855789 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:43.855797 (+ 8us) write_op.cc:312] APPLY: finished
0112 02:52:43.855843 (+ 46us) log.cc:844] Serialized 29 byte log entry
0112 02:52:43.855997 (+ 154us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:43.856010 (+ 13us) write_op.cc:454] Released schema lock
0112 02:52:43.856015 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:43.856358 (+ 343us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:43.856362 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":687,"apply.run_cpu_time_us":310,"apply.run_wall_time_us":762,"num_ops":2,"prepare.queue_time_us":5996,"prepare.run_cpu_time_us":499,"prepare.run_wall_time_us":2676,"replication_time_us":72281}]]}
I20250112 02:52:44.058820 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 83c04f1a2cb04950a9f6170e252c19b4 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:44.059825 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a7717cdcb8a4f47b11ba7af54b57e3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:43 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.060024 2959 tablet_replica.cc:331] T 1a7717cdcb8a4f47b11ba7af54b57e3c P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:44.060055 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a7717cdcb8a4f47b11ba7af54b57e3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:43 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:44.060173 2959 raft_consensus.cc:2238] T 1a7717cdcb8a4f47b11ba7af54b57e3c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:44.060307 2959 raft_consensus.cc:2267] T 1a7717cdcb8a4f47b11ba7af54b57e3c P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:44.060508 2024 tablet_replica.cc:331] T 1a7717cdcb8a4f47b11ba7af54b57e3c P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:44.060665 2024 raft_consensus.cc:2238] T 1a7717cdcb8a4f47b11ba7af54b57e3c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:44.057468 3000 raft_consensus.cc:491] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.060775 2024 raft_consensus.cc:2267] T 1a7717cdcb8a4f47b11ba7af54b57e3c P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:44.060814 3000 raft_consensus.cc:513] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.061235 2959 ts_tablet_manager.cc:1905] T 1a7717cdcb8a4f47b11ba7af54b57e3c P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:44.061312 3000 leader_election.cc:290] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.061342 2024 ts_tablet_manager.cc:1905] T 1a7717cdcb8a4f47b11ba7af54b57e3c P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:44.059389 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a7717cdcb8a4f47b11ba7af54b57e3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:43 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.061509 3028 tablet_replica.cc:331] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:44.061666 3028 raft_consensus.cc:2238] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:44.061787 3028 raft_consensus.cc:2267] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:44.062115 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eed0a9f8354e5391ec2463dafdb683" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:44.062266 3028 ts_tablet_manager.cc:1905] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:44.062604 854 leader_election.cc:343] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.062985 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eed0a9f8354e5391ec2463dafdb683" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:44.063367 855 leader_election.cc:343] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.063450 855 leader_election.cc:304] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.063810 3000 raft_consensus.cc:2743] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.066850 2024 ts_tablet_manager.cc:1918] T 1a7717cdcb8a4f47b11ba7af54b57e3c P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:44.066972 2024 log.cc:1198] T 1a7717cdcb8a4f47b11ba7af54b57e3c P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/1a7717cdcb8a4f47b11ba7af54b57e3c
I20250112 02:52:44.067270 2024 ts_tablet_manager.cc:1939] T 1a7717cdcb8a4f47b11ba7af54b57e3c P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:44.067968 2959 ts_tablet_manager.cc:1918] T 1a7717cdcb8a4f47b11ba7af54b57e3c P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:44.068176 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 1a7717cdcb8a4f47b11ba7af54b57e3c (table default.Table_f7fd6e154c104f7ab66a01e7f8202da6 [id=4447b2d49c864e268d7eec91384899cc]) successfully deleted
I20250112 02:52:44.068706 3028 ts_tablet_manager.cc:1918] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:44.068876 3028 log.cc:1198] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/1a7717cdcb8a4f47b11ba7af54b57e3c
I20250112 02:52:44.069244 3028 ts_tablet_manager.cc:1939] T 1a7717cdcb8a4f47b11ba7af54b57e3c P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:44.069664 448 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:40610 (request call id 199) took 2424 ms. Trace:
I20250112 02:52:44.069777 448 rpcz_store.cc:276] 0112 02:52:41.644921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.058750 (+2413829us) service_pool.cc:225] Handling call
0112 02:52:44.069632 (+ 10882us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:52:44.070075 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_498525c117bf4424a217c15eda6ba3ce" } modify_external_catalogs: true
I20250112 02:52:44.070282 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_498525c117bf4424a217c15eda6ba3ce" } modify_external_catalogs: true
I20250112 02:52:44.070703 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 1a7717cdcb8a4f47b11ba7af54b57e3c (table default.Table_f7fd6e154c104f7ab66a01e7f8202da6 [id=4447b2d49c864e268d7eec91384899cc]) successfully deleted
I20250112 02:52:44.074712 454 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:44.075196 454 catalog_manager.cc:5526] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:44.075820 454 catalog_manager.cc:5526] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:44.076287 454 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:44.076941 454 catalog_manager.cc:5526] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:44.077407 454 catalog_manager.cc:5526] T ea18666ec324403b9f10a099688438b8 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:44.077844 454 catalog_manager.cc:5526] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:44.078799 454 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:44.079331 454 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]). Expected version 2 got 0
I20250112 02:52:44.079604 454 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:44.080153 454 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:44.080382 1177 tablet_bootstrap.cc:654] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.081441 1173 tablet_bootstrap.cc:492] T 5a3af0ea4e954053ae89814f3e74e6b7 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:44.091701 2179 raft_consensus.cc:491] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.091809 2179 raft_consensus.cc:513] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.092317 2179 leader_election.cc:290] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:44.093077 714 leader_election.cc:343] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 8429a3d8636d4e2ca4c1191f85b9266b
W20250112 02:52:44.095593 713 leader_election.cc:343] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 8429a3d8636d4e2ca4c1191f85b9266b
I20250112 02:52:44.095682 713 leader_election.cc:304] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:44.096067 2445 raft_consensus.cc:2743] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.098691 1181 tablet_bootstrap.cc:492] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:44.098906 1181 ts_tablet_manager.cc:1397] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.000s
I20250112 02:52:44.099810 1181 raft_consensus.cc:357] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.100077 1181 raft_consensus.cc:383] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.100174 1181 raft_consensus.cc:738] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.100481 1181 consensus_queue.cc:260] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.104800 1181 ts_tablet_manager.cc:1428] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250112 02:52:44.124064 2978 raft_consensus.cc:491] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.124336 2978 raft_consensus.cc:513] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.125679 1173 tablet_bootstrap.cc:654] T 5a3af0ea4e954053ae89814f3e74e6b7 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.125921 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ffdc237f8640f0b378809f93fc6292" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:44.126087 772 raft_consensus.cc:2463] T a3ffdc237f8640f0b378809f93fc6292 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:44.126668 991 leader_election.cc:304] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:44.126878 2228 raft_consensus.cc:2798] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:44.126930 2228 raft_consensus.cc:491] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:44.126978 2228 raft_consensus.cc:3054] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.129377 2228 raft_consensus.cc:513] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.135754 1177 tablet_bootstrap.cc:492] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:44.135872 1177 ts_tablet_manager.cc:1397] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.086s user 0.000s sys 0.003s
W20250112 02:52:44.136474 989 leader_election.cc:343] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: a3ffdc237f8640f0b378809f93fc6292
I20250112 02:52:44.136559 1177 raft_consensus.cc:357] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.136710 1177 raft_consensus.cc:383] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.136760 1177 raft_consensus.cc:738] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.136955 1177 consensus_queue.cc:260] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.137499 1177 ts_tablet_manager.cc:1428] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:44.139585 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ffdc237f8640f0b378809f93fc6292" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:44.139736 1177 tablet_bootstrap.cc:492] T 62f4bd2df0ce45dabc55373f1154d3ac P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:44.139750 772 raft_consensus.cc:3054] T a3ffdc237f8640f0b378809f93fc6292 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.141027 1177 tablet_bootstrap.cc:654] T 62f4bd2df0ce45dabc55373f1154d3ac P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.142360 772 raft_consensus.cc:2463] T a3ffdc237f8640f0b378809f93fc6292 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:44.143123 991 leader_election.cc:304] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:44.143399 2745 raft_consensus.cc:2798] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:44.143543 2745 raft_consensus.cc:695] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:44.143779 2745 consensus_queue.cc:237] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.145161 2228 leader_election.cc:290] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:44.146339 989 leader_election.cc:343] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: a3ffdc237f8640f0b378809f93fc6292
I20250112 02:52:44.146664 2978 leader_election.cc:290] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:44.126339 2787 raft_consensus.cc:491] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.152653 2787 raft_consensus.cc:513] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.153169 2787 leader_election.cc:290] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.128381 3000 raft_consensus.cc:491] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.153311 3000 raft_consensus.cc:513] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.153767 3000 leader_election.cc:290] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.154445 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93aba9b4d1dd485cb62c4827c916a1d5" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:44.155138 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5079f87652a4ea58b366d1f9448fd4e" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:44.155172 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93aba9b4d1dd485cb62c4827c916a1d5" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:44.155442 855 leader_election.cc:343] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.155714 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5079f87652a4ea58b366d1f9448fd4e" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:44.155864 854 leader_election.cc:343] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:44.156059 854 leader_election.cc:343] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.156139 854 leader_election.cc:304] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:44.156154 855 leader_election.cc:343] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.156253 855 leader_election.cc:304] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.156600 3000 raft_consensus.cc:2743] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.156611 2787 raft_consensus.cc:2743] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.162739 2978 raft_consensus.cc:491] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.162904 2978 raft_consensus.cc:513] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.164155 2978 leader_election.cc:290] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:44.164258 3045 tablet.cc:1721] T fd7394b1878447a189ce45957f10e4df P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:44.164906 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61eac6b572ff47c185768bee7490edf7" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:44.168346 917 raft_consensus.cc:2463] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:44.169217 989 leader_election.cc:304] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:44.169242 2959 log.cc:1198] T 1a7717cdcb8a4f47b11ba7af54b57e3c P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/1a7717cdcb8a4f47b11ba7af54b57e3c
I20250112 02:52:44.169561 2959 ts_tablet_manager.cc:1939] T 1a7717cdcb8a4f47b11ba7af54b57e3c P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:44.170751 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 1a7717cdcb8a4f47b11ba7af54b57e3c (table default.Table_f7fd6e154c104f7ab66a01e7f8202da6 [id=4447b2d49c864e268d7eec91384899cc]) successfully deleted
I20250112 02:52:44.171967 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61eac6b572ff47c185768bee7490edf7" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:44.172160 772 raft_consensus.cc:2463] T 61eac6b572ff47c185768bee7490edf7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:44.173019 2978 raft_consensus.cc:2798] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:44.173132 2978 raft_consensus.cc:491] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:44.173224 2978 raft_consensus.cc:3054] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.175585 2978 raft_consensus.cc:513] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.175889 2787 raft_consensus.cc:491] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.175985 2787 raft_consensus.cc:513] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.176503 2787 leader_election.cc:290] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:44.177224 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24ebf85f76f44633a20d97964638c0ac" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:44.177743 855 leader_election.cc:343] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:44.177937 854 leader_election.cc:343] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 24ebf85f76f44633a20d97964638c0ac
I20250112 02:52:44.178004 854 leader_election.cc:304] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.178330 2787 raft_consensus.cc:2743] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.180972 760 tablet_service.cc:1467] Processing CreateTablet for tablet f92b47749e3348d4ad08ff18b6b266b9 (DEFAULT_TABLE table=default.Table_aca3d1a26d2049e5a0c44926b28dcfcc [id=5963db84ab9d413e9e9021c2786b7977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:44.181311 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f92b47749e3348d4ad08ff18b6b266b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:44.183732 890 tablet_service.cc:1467] Processing CreateTablet for tablet 154ecbf4175a42a7ad765780c0f902a5 (DEFAULT_TABLE table=default.Table_ad5f01f6a5274cc483c8f051059d36c3 [id=8be34ef1e0c54f7fa38ec57fb432b3db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:44.183812 743 tablet_service.cc:1467] Processing CreateTablet for tablet 154ecbf4175a42a7ad765780c0f902a5 (DEFAULT_TABLE table=default.Table_ad5f01f6a5274cc483c8f051059d36c3 [id=8be34ef1e0c54f7fa38ec57fb432b3db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:44.184032 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 154ecbf4175a42a7ad765780c0f902a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:44.184127 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 154ecbf4175a42a7ad765780c0f902a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:44.187781 895 tablet_service.cc:1467] Processing CreateTablet for tablet f92b47749e3348d4ad08ff18b6b266b9 (DEFAULT_TABLE table=default.Table_aca3d1a26d2049e5a0c44926b28dcfcc [id=5963db84ab9d413e9e9021c2786b7977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:44.188061 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f92b47749e3348d4ad08ff18b6b266b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:44.180071 1017 tablet_service.cc:1467] Processing CreateTablet for tablet f92b47749e3348d4ad08ff18b6b266b9 (DEFAULT_TABLE table=default.Table_aca3d1a26d2049e5a0c44926b28dcfcc [id=5963db84ab9d413e9e9021c2786b7977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:44.190838 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f92b47749e3348d4ad08ff18b6b266b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:44.182112 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 154ecbf4175a42a7ad765780c0f902a5 (DEFAULT_TABLE table=default.Table_ad5f01f6a5274cc483c8f051059d36c3 [id=8be34ef1e0c54f7fa38ec57fb432b3db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:44.194890 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 154ecbf4175a42a7ad765780c0f902a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:44.156844 1181 tablet_bootstrap.cc:492] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:44.207291 1181 tablet_bootstrap.cc:654] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.207744 3000 raft_consensus.cc:491] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.207926 3000 raft_consensus.cc:513] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.208701 3000 leader_election.cc:290] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.209375 2985 raft_consensus.cc:491] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.209530 2985 raft_consensus.cc:513] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.210060 2985 leader_election.cc:290] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:44.210683 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54215c55966a46e1ab96e352c4e3764e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:44.210907 776 raft_consensus.cc:2463] T 54215c55966a46e1ab96e352c4e3764e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:44.211393 991 leader_election.cc:304] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:44.211761 2985 raft_consensus.cc:2798] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:44.211838 2985 raft_consensus.cc:491] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:44.211905 2985 raft_consensus.cc:3054] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.214831 2985 raft_consensus.cc:513] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.215696 3000 raft_consensus.cc:491] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.215798 3000 raft_consensus.cc:513] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.216151 1177 tablet_bootstrap.cc:492] T 62f4bd2df0ce45dabc55373f1154d3ac P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:44.216239 3000 leader_election.cc:290] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.216284 1177 ts_tablet_manager.cc:1397] T 62f4bd2df0ce45dabc55373f1154d3ac P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.077s user 0.004s sys 0.005s
I20250112 02:52:44.216898 1177 raft_consensus.cc:357] T 62f4bd2df0ce45dabc55373f1154d3ac P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.216984 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76009bac22ce4136b148b493a004a0f7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:44.217067 1177 raft_consensus.cc:383] T 62f4bd2df0ce45dabc55373f1154d3ac P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.217110 1177 raft_consensus.cc:738] T 62f4bd2df0ce45dabc55373f1154d3ac P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.217278 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fea8f6f13d54945934ee3880e00c5e9" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:44.217320 1177 consensus_queue.cc:260] T 62f4bd2df0ce45dabc55373f1154d3ac P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:44.218065 855 leader_election.cc:343] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:44.218284 855 leader_election.cc:343] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.218884 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54215c55966a46e1ab96e352c4e3764e" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:44.219066 776 raft_consensus.cc:3054] T 54215c55966a46e1ab96e352c4e3764e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.221438 776 raft_consensus.cc:2463] T 54215c55966a46e1ab96e352c4e3764e P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:44.222375 991 leader_election.cc:304] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
W20250112 02:52:44.224081 989 leader_election.cc:343] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 54215c55966a46e1ab96e352c4e3764e
W20250112 02:52:44.224467 989 leader_election.cc:343] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 54215c55966a46e1ab96e352c4e3764e
I20250112 02:52:44.224692 1177 ts_tablet_manager.cc:1428] T 62f4bd2df0ce45dabc55373f1154d3ac P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250112 02:52:44.224853 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fea8f6f13d54945934ee3880e00c5e9" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:44.224962 1177 tablet_bootstrap.cc:492] T 9b81aee21510452881948e41d7aba9be P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:44.225032 2228 raft_consensus.cc:2798] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:44.225239 2228 raft_consensus.cc:695] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:44.225495 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76009bac22ce4136b148b493a004a0f7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:44.225908 854 leader_election.cc:343] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.226001 854 leader_election.cc:304] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.225471 2228 consensus_queue.cc:237] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.226305 1177 tablet_bootstrap.cc:654] T 9b81aee21510452881948e41d7aba9be P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.227537 2787 raft_consensus.cc:491] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.227649 2787 raft_consensus.cc:513] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.228137 2787 leader_election.cc:290] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:44.228783 854 leader_election.cc:343] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.228904 854 leader_election.cc:304] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.229099 2673 raft_consensus.cc:2743] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.229382 2673 raft_consensus.cc:2743] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.230446 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d7774d4a284d24b833d50809996fc2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:44.230924 854 leader_election.cc:343] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.233201 1181 tablet_bootstrap.cc:492] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:44.234303 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d7774d4a284d24b833d50809996fc2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:44.234918 855 leader_election.cc:343] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.235157 855 leader_election.cc:304] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.235555 3000 raft_consensus.cc:2743] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.235795 1181 ts_tablet_manager.cc:1397] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.079s user 0.001s sys 0.002s
W20250112 02:52:44.236055 991 consensus_peers.cc:487] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:44.236574 1181 raft_consensus.cc:357] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.236732 1181 raft_consensus.cc:383] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.236775 1181 raft_consensus.cc:738] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.236958 1181 consensus_queue.cc:260] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.215436 2985 leader_election.cc:290] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:44.241873 1181 ts_tablet_manager.cc:1428] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250112 02:52:44.242120 1181 tablet_bootstrap.cc:492] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:44.244416 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61eac6b572ff47c185768bee7490edf7" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:44.244608 772 raft_consensus.cc:3054] T 61eac6b572ff47c185768bee7490edf7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.247027 1181 tablet_bootstrap.cc:654] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.248080 772 raft_consensus.cc:2463] T 61eac6b572ff47c185768bee7490edf7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:44.252229 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61eac6b572ff47c185768bee7490edf7" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:44.252418 917 raft_consensus.cc:3054] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.255617 917 raft_consensus.cc:2463] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:44.256050 989 leader_election.cc:304] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:44.256323 2985 raft_consensus.cc:2798] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:44.256444 2985 raft_consensus.cc:695] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:44.257611 2978 leader_election.cc:290] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:44.256705 2985 consensus_queue.cc:237] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.266686 3000 raft_consensus.cc:491] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.266799 3000 raft_consensus.cc:513] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.267323 3000 leader_election.cc:290] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
W20250112 02:52:44.268190 854 leader_election.cc:343] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 896908b2726c40f9aabdfc9a6b3283f7
I20250112 02:52:44.268502 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896908b2726c40f9aabdfc9a6b3283f7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:44.268921 855 leader_election.cc:343] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.269012 855 leader_election.cc:304] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.269299 3000 raft_consensus.cc:2743] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.287266 3044 tablet.cc:1721] T fd7394b1878447a189ce45957f10e4df P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:44.287962 3000 consensus_queue.cc:1035] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:44.291774 3045 tablet.cc:1721] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:44.295243 2179 raft_consensus.cc:491] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.295367 2179 raft_consensus.cc:513] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:44.297358 713 leader_election.cc:343] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 7e1954e7859f422f9690fe4a9384a8bb
I20250112 02:52:44.297570 2179 leader_election.cc:290] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:44.302853 2179 raft_consensus.cc:491] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.302965 2179 raft_consensus.cc:513] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.304948 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e1954e7859f422f9690fe4a9384a8bb" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:44.305122 1051 raft_consensus.cc:2463] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
W20250112 02:52:44.305159 713 leader_election.cc:343] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: d63d411f906b4d4eb68343bdcc493e05
I20250112 02:52:44.305586 714 leader_election.cc:304] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:44.305805 2451 raft_consensus.cc:2798] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:44.305868 2451 raft_consensus.cc:491] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:44.305917 2451 raft_consensus.cc:3054] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.308523 2451 raft_consensus.cc:513] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.308727 1181 tablet_bootstrap.cc:492] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:44.308842 1177 tablet_bootstrap.cc:492] T 9b81aee21510452881948e41d7aba9be P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:44.308854 1181 ts_tablet_manager.cc:1397] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.067s user 0.002s sys 0.015s
I20250112 02:52:44.308960 1177 ts_tablet_manager.cc:1397] T 9b81aee21510452881948e41d7aba9be P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.003s
I20250112 02:52:44.309626 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e1954e7859f422f9690fe4a9384a8bb" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:44.309665 1181 raft_consensus.cc:357] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.309779 1051 raft_consensus.cc:3054] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.309827 1181 raft_consensus.cc:383] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.309888 1181 raft_consensus.cc:738] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
W20250112 02:52:44.310029 713 leader_election.cc:343] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 7e1954e7859f422f9690fe4a9384a8bb
I20250112 02:52:44.309732 2451 leader_election.cc:290] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:44.310909 1177 raft_consensus.cc:357] T 9b81aee21510452881948e41d7aba9be P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.311077 1177 raft_consensus.cc:383] T 9b81aee21510452881948e41d7aba9be P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.311134 1177 raft_consensus.cc:738] T 9b81aee21510452881948e41d7aba9be P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.311151 1181 consensus_queue.cc:260] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.310438 3058 tablet.cc:1721] T 79eb61cedd544c49869471eebe2fd237 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:44.311398 1177 consensus_queue.cc:260] T 9b81aee21510452881948e41d7aba9be P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.311863 1177 ts_tablet_manager.cc:1428] T 9b81aee21510452881948e41d7aba9be P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:44.311990 1181 ts_tablet_manager.cc:1428] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 02:52:44.312103 1177 tablet_bootstrap.cc:492] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:44.312223 1181 tablet_bootstrap.cc:492] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:44.315423 1181 tablet_bootstrap.cc:654] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.315665 1051 raft_consensus.cc:2463] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:44.315980 714 leader_election.cc:304] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:44.317453 1177 tablet_bootstrap.cc:654] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.321538 2451 raft_consensus.cc:2798] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:44.321995 2451 raft_consensus.cc:695] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:44.322350 2451 consensus_queue.cc:237] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.329319 1181 tablet_bootstrap.cc:492] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:44.329442 1181 ts_tablet_manager.cc:1397] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.005s
I20250112 02:52:44.313587 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63d411f906b4d4eb68343bdcc493e05" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:44.330018 1052 raft_consensus.cc:2463] T d63d411f906b4d4eb68343bdcc493e05 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:44.330588 714 leader_election.cc:304] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:44.330855 2445 raft_consensus.cc:2798] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:44.330919 2445 raft_consensus.cc:491] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:44.330973 2445 raft_consensus.cc:3054] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.333377 2445 raft_consensus.cc:513] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.334075 2445 leader_election.cc:290] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:44.334024 1181 raft_consensus.cc:357] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.334174 1181 raft_consensus.cc:383] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.334215 1181 raft_consensus.cc:738] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.335045 1181 consensus_queue.cc:260] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:44.335590 713 leader_election.cc:343] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: d63d411f906b4d4eb68343bdcc493e05
I20250112 02:52:44.336548 1173 tablet_bootstrap.cc:492] T 5a3af0ea4e954053ae89814f3e74e6b7 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:44.336679 1173 ts_tablet_manager.cc:1397] T 5a3af0ea4e954053ae89814f3e74e6b7 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.255s user 0.016s sys 0.019s
I20250112 02:52:44.335908 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63d411f906b4d4eb68343bdcc493e05" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:44.336920 1052 raft_consensus.cc:3054] T d63d411f906b4d4eb68343bdcc493e05 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.337549 2179 leader_election.cc:290] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:44.338665 1173 raft_consensus.cc:357] T 5a3af0ea4e954053ae89814f3e74e6b7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.338832 1173 raft_consensus.cc:383] T 5a3af0ea4e954053ae89814f3e74e6b7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.338876 1173 raft_consensus.cc:738] T 5a3af0ea4e954053ae89814f3e74e6b7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.339030 1173 consensus_queue.cc:260] T 5a3af0ea4e954053ae89814f3e74e6b7 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.339489 1173 ts_tablet_manager.cc:1428] T 5a3af0ea4e954053ae89814f3e74e6b7 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 02:52:44.339701 1173 tablet_bootstrap.cc:492] T 26577b614e32458e8bc5d6830c8c2e40 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:44.342483 1173 tablet_bootstrap.cc:654] T 26577b614e32458e8bc5d6830c8c2e40 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.343533 2673 consensus_queue.cc:1035] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:44.347257 1052 raft_consensus.cc:2463] T d63d411f906b4d4eb68343bdcc493e05 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:44.347666 714 leader_election.cc:304] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:44.348352 3050 tablet.cc:1721] T fd7394b1878447a189ce45957f10e4df P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:44.350354 2179 raft_consensus.cc:2798] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:44.350679 2179 raft_consensus.cc:695] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:44.351117 2179 consensus_queue.cc:237] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.357409 1177 tablet_bootstrap.cc:492] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:44.357734 1177 ts_tablet_manager.cc:1397] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.000s
I20250112 02:52:44.358345 1177 raft_consensus.cc:357] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.358559 1177 raft_consensus.cc:383] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.358641 1177 raft_consensus.cc:738] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.358839 1177 consensus_queue.cc:260] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.359264 1177 ts_tablet_manager.cc:1428] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250112 02:52:44.359537 1177 tablet_bootstrap.cc:492] T 164f76edcf8f4bd498dcd2c25fd7477d P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:44.361013 1177 tablet_bootstrap.cc:654] T 164f76edcf8f4bd498dcd2c25fd7477d P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.368744 2673 raft_consensus.cc:491] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.368863 2673 raft_consensus.cc:513] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.369431 2673 leader_election.cc:290] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.369836 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1123077650c4e89bffddd6d1264e428" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:44.369968 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1123077650c4e89bffddd6d1264e428" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:44.370230 854 leader_election.cc:343] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:44.370395 855 leader_election.cc:343] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.370487 855 leader_election.cc:304] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.370967 2673 raft_consensus.cc:2743] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.371228 1173 tablet_bootstrap.cc:492] T 26577b614e32458e8bc5d6830c8c2e40 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:44.371349 1173 ts_tablet_manager.cc:1397] T 26577b614e32458e8bc5d6830c8c2e40 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.001s
I20250112 02:52:44.371961 1177 tablet_bootstrap.cc:492] T 164f76edcf8f4bd498dcd2c25fd7477d P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:44.371949 1173 raft_consensus.cc:357] T 26577b614e32458e8bc5d6830c8c2e40 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.372115 1173 raft_consensus.cc:383] T 26577b614e32458e8bc5d6830c8c2e40 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.372169 1173 raft_consensus.cc:738] T 26577b614e32458e8bc5d6830c8c2e40 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.372278 1177 ts_tablet_manager.cc:1397] T 164f76edcf8f4bd498dcd2c25fd7477d P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20250112 02:52:44.372356 1173 consensus_queue.cc:260] T 26577b614e32458e8bc5d6830c8c2e40 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.372994 1173 ts_tablet_manager.cc:1428] T 26577b614e32458e8bc5d6830c8c2e40 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 02:52:44.373301 1173 tablet_bootstrap.cc:492] T fd81852962fb43449e35ef45dc1daaad P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:44.373157 1177 raft_consensus.cc:357] T 164f76edcf8f4bd498dcd2c25fd7477d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.373487 1177 raft_consensus.cc:383] T 164f76edcf8f4bd498dcd2c25fd7477d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.373581 1177 raft_consensus.cc:738] T 164f76edcf8f4bd498dcd2c25fd7477d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.373785 1177 consensus_queue.cc:260] T 164f76edcf8f4bd498dcd2c25fd7477d P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.374171 1177 ts_tablet_manager.cc:1428] T 164f76edcf8f4bd498dcd2c25fd7477d P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:44.374467 1177 tablet_bootstrap.cc:492] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:44.375197 1173 tablet_bootstrap.cc:654] T fd81852962fb43449e35ef45dc1daaad P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.375186 1181 ts_tablet_manager.cc:1428] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.046s user 0.000s sys 0.002s
I20250112 02:52:44.376039 3050 tablet.cc:1721] T 79eb61cedd544c49869471eebe2fd237 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:44.376536 1177 tablet_bootstrap.cc:654] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.381742 1181 tablet_bootstrap.cc:492] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:44.383256 1181 tablet_bootstrap.cc:654] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.403689 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3dab0466476c4a9aa265271c1abd91a4 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:44.415510 2673 consensus_queue.cc:1035] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:44.418447 2985 consensus_queue.cc:1035] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:44.427637 2228 consensus_queue.cc:1035] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:44.427711 2179 raft_consensus.cc:491] T 709517ae520640208abe85fa14a5301e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.427831 2179 raft_consensus.cc:513] T 709517ae520640208abe85fa14a5301e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.428385 2179 leader_election.cc:290] T 709517ae520640208abe85fa14a5301e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:44.429169 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709517ae520640208abe85fa14a5301e" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:44.429669 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu pid 1229
I20250112 02:52:44.431152 2985 consensus_queue.cc:1035] T 03a32b424ec4431d88f6af45dd09089f P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:44.434830 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709517ae520640208abe85fa14a5301e" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:44.435334 713 leader_election.cc:304] T 709517ae520640208abe85fa14a5301e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:44.435858 2451 raft_consensus.cc:2743] T 709517ae520640208abe85fa14a5301e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.443321 776 raft_consensus.cc:3054] T 709517ae520640208abe85fa14a5301e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.446548 2977 consensus_queue.cc:1035] T 709517ae520640208abe85fa14a5301e P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:44.457703 1177 tablet_bootstrap.cc:492] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:44.458961 1177 ts_tablet_manager.cc:1397] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.085s user 0.011s sys 0.010s
I20250112 02:52:44.459349 1173 tablet_bootstrap.cc:492] T fd81852962fb43449e35ef45dc1daaad P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:44.459486 1173 ts_tablet_manager.cc:1397] T fd81852962fb43449e35ef45dc1daaad P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.086s user 0.001s sys 0.019s
I20250112 02:52:44.460091 1173 raft_consensus.cc:357] T fd81852962fb43449e35ef45dc1daaad P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.460091 1177 raft_consensus.cc:357] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.460243 1173 raft_consensus.cc:383] T fd81852962fb43449e35ef45dc1daaad P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.460289 1173 raft_consensus.cc:738] T fd81852962fb43449e35ef45dc1daaad P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.460486 1173 consensus_queue.cc:260] T fd81852962fb43449e35ef45dc1daaad P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.461095 1173 ts_tablet_manager.cc:1428] T fd81852962fb43449e35ef45dc1daaad P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:44.461357 1173 tablet_bootstrap.cc:492] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:44.461724 1177 raft_consensus.cc:383] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.461980 1177 raft_consensus.cc:738] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.462574 1177 consensus_queue.cc:260] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.463163 1173 tablet_bootstrap.cc:654] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.463117 2673 raft_consensus.cc:491] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.463238 2673 raft_consensus.cc:513] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.463923 2673 leader_election.cc:290] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.464525 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5804fc4130f84c69832c1022b7bd7fa5" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:44.465363 855 leader_election.cc:343] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.466120 1177 ts_tablet_manager.cc:1428] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
W20250112 02:52:44.466389 854 leader_election.cc:343] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 5804fc4130f84c69832c1022b7bd7fa5
I20250112 02:52:44.466504 854 leader_election.cc:304] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.466711 2673 raft_consensus.cc:2743] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.471583 2673 raft_consensus.cc:491] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.471690 2673 raft_consensus.cc:513] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.472190 2673 leader_election.cc:290] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:44.477569 2745 raft_consensus.cc:491] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.477679 2745 raft_consensus.cc:513] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.478116 2745 leader_election.cc:290] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:44.479987 854 leader_election.cc:343] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 164f76edcf8f4bd498dcd2c25fd7477d
W20250112 02:52:44.480296 989 leader_election.cc:343] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 62f4bd2df0ce45dabc55373f1154d3ac
I20250112 02:52:44.483671 1426 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 02:52:44.472739 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164f76edcf8f4bd498dcd2c25fd7477d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:44.486546 779 raft_consensus.cc:2463] T 164f76edcf8f4bd498dcd2c25fd7477d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:44.479449 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f4bd2df0ce45dabc55373f1154d3ac" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:44.486742 778 raft_consensus.cc:2463] T 62f4bd2df0ce45dabc55373f1154d3ac P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:44.487043 855 leader_election.cc:304] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.487146 991 leader_election.cc:304] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:44.487396 3000 raft_consensus.cc:2798] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:44.487457 3000 raft_consensus.cc:491] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:44.487529 3000 raft_consensus.cc:3054] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.487854 2745 raft_consensus.cc:2798] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:44.487919 2745 raft_consensus.cc:491] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:44.487965 2745 raft_consensus.cc:3054] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.490167 1177 tablet_bootstrap.cc:492] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:44.491997 1052 raft_consensus.cc:3054] T 26577b614e32458e8bc5d6830c8c2e40 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.492697 1177 tablet_bootstrap.cc:654] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.499527 2787 raft_consensus.cc:491] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.499672 2787 raft_consensus.cc:513] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.500207 2787 leader_election.cc:290] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.500550 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b741e4f6abd4695b3ba1d7a667b6046" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:44.500666 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b741e4f6abd4695b3ba1d7a667b6046" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:44.501022 854 leader_election.cc:343] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:44.501180 855 leader_election.cc:343] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.501284 854 leader_election.cc:304] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.501508 2787 raft_consensus.cc:2743] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.505009 2977 consensus_queue.cc:1035] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:44.506858 2787 raft_consensus.cc:491] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.506968 2787 raft_consensus.cc:513] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.507503 2787 leader_election.cc:290] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.508359 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3d291ca3494f05ac23c7fe293e0bd9" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:44.508814 855 leader_election.cc:343] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.509091 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3d291ca3494f05ac23c7fe293e0bd9" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:44.509228 1173 tablet_bootstrap.cc:492] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:44.509358 1173 ts_tablet_manager.cc:1397] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.000s
W20250112 02:52:44.509861 854 leader_election.cc:343] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.509990 854 leader_election.cc:304] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.510224 2787 raft_consensus.cc:2743] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.514910 1177 tablet_bootstrap.cc:492] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:44.510083 1173 raft_consensus.cc:357] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.515658 1173 raft_consensus.cc:383] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.515733 1173 raft_consensus.cc:738] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.515983 1173 consensus_queue.cc:260] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.516249 1177 ts_tablet_manager.cc:1397] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.002s
I20250112 02:52:44.516618 1173 ts_tablet_manager.cc:1428] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250112 02:52:44.516870 1173 tablet_bootstrap.cc:492] T c5079f87652a4ea58b366d1f9448fd4e P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:44.517267 1177 raft_consensus.cc:357] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.518347 1177 raft_consensus.cc:383] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.518455 1177 raft_consensus.cc:738] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.518694 1181 tablet_bootstrap.cc:492] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:44.518846 1181 ts_tablet_manager.cc:1397] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.137s user 0.009s sys 0.015s
I20250112 02:52:44.519174 3000 raft_consensus.cc:513] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.518703 1177 consensus_queue.cc:260] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.519919 1181 raft_consensus.cc:357] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.520536 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164f76edcf8f4bd498dcd2c25fd7477d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:44.520073 1177 ts_tablet_manager.cc:1428] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 02:52:44.520713 778 raft_consensus.cc:3054] T 164f76edcf8f4bd498dcd2c25fd7477d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.520926 1177 tablet_bootstrap.cc:492] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:44.522365 1177 tablet_bootstrap.cc:654] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.520023 3000 leader_election.cc:290] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:44.523305 1181 raft_consensus.cc:383] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.523411 1181 raft_consensus.cc:738] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.523811 1181 consensus_queue.cc:260] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.524031 3000 raft_consensus.cc:491] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.524124 3000 raft_consensus.cc:513] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.524622 3000 leader_election.cc:290] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.525395 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1ae3f34b434ee5bc3519cce8d283a6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:44.525573 1181 ts_tablet_manager.cc:1428] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250112 02:52:44.525804 1181 tablet_bootstrap.cc:492] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
W20250112 02:52:44.525853 855 leader_election.cc:343] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.528821 1173 tablet_bootstrap.cc:654] T c5079f87652a4ea58b366d1f9448fd4e P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.532931 1181 tablet_bootstrap.cc:654] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.533847 2745 raft_consensus.cc:513] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.535075 2745 leader_election.cc:290] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:44.535763 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f4bd2df0ce45dabc55373f1154d3ac" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:44.535923 779 raft_consensus.cc:3054] T 62f4bd2df0ce45dabc55373f1154d3ac P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.537596 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1ae3f34b434ee5bc3519cce8d283a6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:44.540129 854 leader_election.cc:343] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.540235 854 leader_election.cc:304] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:44.540619 854 leader_election.cc:343] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 164f76edcf8f4bd498dcd2c25fd7477d
W20250112 02:52:44.540915 989 leader_election.cc:343] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 62f4bd2df0ce45dabc55373f1154d3ac
I20250112 02:52:44.540881 3000 raft_consensus.cc:2743] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.550338 1177 tablet_bootstrap.cc:492] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:44.550540 1177 ts_tablet_manager.cc:1397] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.010s
I20250112 02:52:44.551330 1181 tablet_bootstrap.cc:492] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:44.551250 1177 raft_consensus.cc:357] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.551520 1177 raft_consensus.cc:383] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.551635 1177 raft_consensus.cc:738] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.551978 1177 consensus_queue.cc:260] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.552423 1181 ts_tablet_manager.cc:1397] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.003s
I20250112 02:52:44.552538 1177 ts_tablet_manager.cc:1428] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:44.552836 1177 tablet_bootstrap.cc:492] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:44.553139 1181 raft_consensus.cc:357] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.553522 1181 raft_consensus.cc:383] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.553629 1181 raft_consensus.cc:738] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.553907 1181 consensus_queue.cc:260] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.554522 1181 ts_tablet_manager.cc:1428] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:44.554780 1181 tablet_bootstrap.cc:492] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:44.555068 1177 tablet_bootstrap.cc:654] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.556224 1181 tablet_bootstrap.cc:654] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.560272 1173 tablet_bootstrap.cc:492] T c5079f87652a4ea58b366d1f9448fd4e P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:44.560460 1173 ts_tablet_manager.cc:1397] T c5079f87652a4ea58b366d1f9448fd4e P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.003s
I20250112 02:52:44.561244 1173 raft_consensus.cc:357] T c5079f87652a4ea58b366d1f9448fd4e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.561447 1173 raft_consensus.cc:383] T c5079f87652a4ea58b366d1f9448fd4e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.561542 1173 raft_consensus.cc:738] T c5079f87652a4ea58b366d1f9448fd4e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.561810 1173 consensus_queue.cc:260] T c5079f87652a4ea58b366d1f9448fd4e P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.562333 1173 ts_tablet_manager.cc:1428] T c5079f87652a4ea58b366d1f9448fd4e P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:44.562657 1173 tablet_bootstrap.cc:492] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:44.415606 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:44.566983 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_5d473cabff2e46e0b531fbce28d9d558 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 1 got 0
I20250112 02:52:44.567196 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:44.567656 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:44.569860 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet fd7394b1878447a189ce45957f10e4df (table default.Table_6113c9f3ad094f0381176b06d637966c [id=08a760dcd20642fb869437d4ccc3f333]). Expected version 1 got 0
I20250112 02:52:44.571358 1177 tablet_bootstrap.cc:492] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:44.571542 1177 ts_tablet_manager.cc:1397] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20250112 02:52:44.572176 1177 raft_consensus.cc:357] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.572407 1177 raft_consensus.cc:383] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.572489 1177 raft_consensus.cc:738] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.572739 1177 consensus_queue.cc:260] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.573320 2179 raft_consensus.cc:491] T 9b81aee21510452881948e41d7aba9be P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.573398 1177 ts_tablet_manager.cc:1428] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:44.573428 2179 raft_consensus.cc:513] T 9b81aee21510452881948e41d7aba9be P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.573624 1177 tablet_bootstrap.cc:492] T ca6f080e5fd84084b601b646d4e6c9e2 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:44.573977 2179 leader_election.cc:290] T 9b81aee21510452881948e41d7aba9be P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:44.574440 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b81aee21510452881948e41d7aba9be" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:44.575515 1177 tablet_bootstrap.cc:654] T ca6f080e5fd84084b601b646d4e6c9e2 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.575750 1181 tablet_bootstrap.cc:492] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:44.575917 1181 ts_tablet_manager.cc:1397] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.002s
I20250112 02:52:44.576905 1181 raft_consensus.cc:357] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.577100 1181 raft_consensus.cc:383] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.577178 1181 raft_consensus.cc:738] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.577385 1181 consensus_queue.cc:260] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.577986 1181 ts_tablet_manager.cc:1428] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:44.578244 1181 tablet_bootstrap.cc:492] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:44.579794 1181 tablet_bootstrap.cc:654] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.580893 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b81aee21510452881948e41d7aba9be" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:44.581385 714 leader_election.cc:304] T 9b81aee21510452881948e41d7aba9be P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:44.581751 2179 raft_consensus.cc:2743] T 9b81aee21510452881948e41d7aba9be P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.585426 778 raft_consensus.cc:2463] T 164f76edcf8f4bd498dcd2c25fd7477d P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:44.588383 779 raft_consensus.cc:2463] T 62f4bd2df0ce45dabc55373f1154d3ac P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:44.588897 855 leader_election.cc:304] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.589268 3000 raft_consensus.cc:2798] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:44.589382 3000 raft_consensus.cc:695] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:44.589603 3000 consensus_queue.cc:237] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.590354 991 leader_election.cc:304] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:44.590833 2745 raft_consensus.cc:2798] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:44.590968 2745 raft_consensus.cc:695] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:44.591195 2745 consensus_queue.cc:237] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.591552 1177 tablet_bootstrap.cc:492] T ca6f080e5fd84084b601b646d4e6c9e2 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:44.591769 1177 ts_tablet_manager.cc:1397] T ca6f080e5fd84084b601b646d4e6c9e2 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.002s
I20250112 02:52:44.591892 2787 raft_consensus.cc:491] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.592010 2787 raft_consensus.cc:513] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.592391 1177 raft_consensus.cc:357] T ca6f080e5fd84084b601b646d4e6c9e2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.592522 2787 leader_election.cc:290] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.592550 1177 raft_consensus.cc:383] T ca6f080e5fd84084b601b646d4e6c9e2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.592603 1177 raft_consensus.cc:738] T ca6f080e5fd84084b601b646d4e6c9e2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.592927 1177 consensus_queue.cc:260] T ca6f080e5fd84084b601b646d4e6c9e2 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.593039 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2796289c7e8f4b45b26508c50b57c640" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:44.593472 1177 ts_tablet_manager.cc:1428] T ca6f080e5fd84084b601b646d4e6c9e2 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:44.593740 1177 tablet_bootstrap.cc:492] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:44.593775 855 leader_election.cc:343] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.595826 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_a9907ed8bfe14625977ec98b67e9d599 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 2 got 0
I20250112 02:52:44.597167 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_b9dff6cb7bbd4d0a88de458399b0d2c0 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 2 got 0
I20250112 02:52:44.599363 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2796289c7e8f4b45b26508c50b57c640" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:44.600137 2787 consensus_queue.cc:1035] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:52:44.600162 854 leader_election.cc:343] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.600349 854 leader_election.cc:304] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.600597 1177 tablet_bootstrap.cc:654] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.600725 2787 raft_consensus.cc:2743] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.601056 2673 raft_consensus.cc:491] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.601161 2673 raft_consensus.cc:513] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.601630 2673 leader_election.cc:290] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.602075 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ddc025a8314f6f84d229191eb56692" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:44.602530 855 leader_election.cc:343] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.606878 1173 tablet_bootstrap.cc:654] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.608500 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ddc025a8314f6f84d229191eb56692" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:44.609361 854 leader_election.cc:343] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.609694 854 leader_election.cc:304] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.609905 2673 raft_consensus.cc:491] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.609992 2673 raft_consensus.cc:513] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.610450 2673 leader_election.cc:290] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.610555 2673 raft_consensus.cc:2743] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.611225 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc52271535df47e282bf1625c2e8e041" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:44.611784 855 leader_election.cc:343] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.612201 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc52271535df47e282bf1625c2e8e041" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:44.612730 854 leader_election.cc:343] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.612864 854 leader_election.cc:304] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.613139 2673 raft_consensus.cc:2743] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.616168 3000 raft_consensus.cc:491] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.616304 3000 raft_consensus.cc:513] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.617130 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99407ead9cf547e3bbe78c0d0b327e6a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:44.617568 855 leader_election.cc:343] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.617884 3000 leader_election.cc:290] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.618402 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99407ead9cf547e3bbe78c0d0b327e6a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:44.618935 854 leader_election.cc:343] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.619079 854 leader_election.cc:304] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.619421 3000 raft_consensus.cc:2743] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.597508 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ca2165362ee541adac724b63f39b1489 (table default.Table_41b7056b6e9240928d0af52aec2e9338 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 2 got 0
I20250112 02:52:44.624804 1181 tablet_bootstrap.cc:492] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:44.624979 1181 ts_tablet_manager.cc:1397] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.000s
I20250112 02:52:44.625865 1181 raft_consensus.cc:357] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.626049 1181 raft_consensus.cc:383] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.626114 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 56ddc025a8314f6f84d229191eb56692 (table default.Table_d22db1cc1d354af495d9888de1bfc22a [id=2ddd5dd296b94bc8b96a97b66e775836]). Expected version 1 got 0
I20250112 02:52:44.626145 1181 raft_consensus.cc:738] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.626408 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:44.626385 1181 consensus_queue.cc:260] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.626947 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:44.626971 1181 ts_tablet_manager.cc:1428] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:44.627224 1181 tablet_bootstrap.cc:492] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:44.628055 1173 tablet_bootstrap.cc:492] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:44.628173 1173 ts_tablet_manager.cc:1397] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.002s
I20250112 02:52:44.628393 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:44.628892 1181 tablet_bootstrap.cc:654] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.628821 1173 raft_consensus.cc:357] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.628973 1173 raft_consensus.cc:383] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.629029 1173 raft_consensus.cc:738] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.629205 1173 consensus_queue.cc:260] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.629688 1173 ts_tablet_manager.cc:1428] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:44.629912 1173 tablet_bootstrap.cc:492] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:44.630127 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:44.630517 1177 tablet_bootstrap.cc:492] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:44.630687 1177 ts_tablet_manager.cc:1397] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.007s
I20250112 02:52:44.631381 1173 tablet_bootstrap.cc:654] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.631291 1177 raft_consensus.cc:357] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.631562 1177 raft_consensus.cc:383] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.631668 1177 raft_consensus.cc:738] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.631908 1177 consensus_queue.cc:260] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.635608 1177 ts_tablet_manager.cc:1428] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 02:52:44.635872 1177 tablet_bootstrap.cc:492] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:44.637260 1177 tablet_bootstrap.cc:654] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.637172 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:44.641837 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5524d75599f04becbe6a94426adb7026 (table default.Table_e8cef38ebaa243ed83bfb41bcea1933f [id=b7739f28885a453d9ef3966511183c17]). Expected version 1 got 0
I20250112 02:52:44.645540 1181 tablet_bootstrap.cc:492] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:44.645692 1181 ts_tablet_manager.cc:1397] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20250112 02:52:44.646368 1181 raft_consensus.cc:357] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.646564 1181 raft_consensus.cc:383] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.646677 1181 raft_consensus.cc:738] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.646952 1181 consensus_queue.cc:260] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.647091 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 37d38f5e525440f8a0f7822620933435 (table default.Table_5aeb2d7bb68c46a589a48526dcc40aaf [id=9d267774bf59428e854651956fbd79be]). Expected version 1 got 0
I20250112 02:52:44.647550 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 1 got 0
I20250112 02:52:44.647629 1181 ts_tablet_manager.cc:1428] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:44.649188 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:44.650465 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet d95f98ae7bdf42518bbff55e257fea88 (table default.Table_2fcdbb05970f4a098f7ba3e287fa0348 [id=70c61753e9944e1fbd65e9ec41a48ee2]). Expected version 1 got 0
I20250112 02:52:44.652012 1181 tablet_bootstrap.cc:492] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:44.653625 1181 tablet_bootstrap.cc:654] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.656458 2445 raft_consensus.cc:491] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.656580 2445 raft_consensus.cc:513] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.657058 2445 leader_election.cc:290] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:44.658156 714 leader_election.cc:343] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: b7340a7a916c4b2fab32e0ba2d0ce322
I20250112 02:52:44.658625 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7340a7a916c4b2fab32e0ba2d0ce322" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:44.658860 917 raft_consensus.cc:2463] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:44.659137 3000 raft_consensus.cc:491] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.659201 713 leader_election.cc:304] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.659276 3000 raft_consensus.cc:513] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.659529 2445 raft_consensus.cc:2798] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:44.659596 2445 raft_consensus.cc:491] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:44.659641 2445 raft_consensus.cc:3054] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.659832 3000 leader_election.cc:290] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:44.660108 1177 tablet_bootstrap.cc:492] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:44.660218 1177 ts_tablet_manager.cc:1397] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.001s
I20250112 02:52:44.660280 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6f080e5fd84084b601b646d4e6c9e2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:44.660434 779 raft_consensus.cc:2463] T ca6f080e5fd84084b601b646d4e6c9e2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:44.660826 1177 raft_consensus.cc:357] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.660902 855 leader_election.cc:304] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:44.660985 1177 raft_consensus.cc:383] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.661031 1177 raft_consensus.cc:738] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.661199 3000 raft_consensus.cc:2798] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:44.661263 3000 raft_consensus.cc:491] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:44.661319 3000 raft_consensus.cc:3054] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.661348 1177 consensus_queue.cc:260] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.661885 1177 ts_tablet_manager.cc:1428] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:44.662097 1177 tablet_bootstrap.cc:492] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:44.662354 854 leader_election.cc:343] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: ca6f080e5fd84084b601b646d4e6c9e2
I20250112 02:52:44.663672 1177 tablet_bootstrap.cc:654] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.664363 1181 tablet_bootstrap.cc:492] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:44.664538 1181 ts_tablet_manager.cc:1397] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250112 02:52:44.665238 1181 raft_consensus.cc:357] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.665433 1181 raft_consensus.cc:383] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.665511 1181 raft_consensus.cc:738] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.665732 1181 consensus_queue.cc:260] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.666288 1181 ts_tablet_manager.cc:1428] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:44.666564 1181 tablet_bootstrap.cc:492] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:44.668067 1181 tablet_bootstrap.cc:654] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.669741 3000 raft_consensus.cc:513] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.670552 3000 leader_election.cc:290] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:44.671159 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6f080e5fd84084b601b646d4e6c9e2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:44.671387 779 raft_consensus.cc:3054] T ca6f080e5fd84084b601b646d4e6c9e2 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:44.672961 854 leader_election.cc:343] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: ca6f080e5fd84084b601b646d4e6c9e2
I20250112 02:52:44.673178 2179 raft_consensus.cc:491] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.673280 2179 raft_consensus.cc:513] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.673768 2179 leader_election.cc:290] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:44.674476 714 leader_election.cc:343] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 5804fc4130f84c69832c1022b7bd7fa5
I20250112 02:52:44.675092 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5804fc4130f84c69832c1022b7bd7fa5" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:44.675243 910 raft_consensus.cc:2463] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:44.676723 713 leader_election.cc:304] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.677151 2451 raft_consensus.cc:2798] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:44.677213 2451 raft_consensus.cc:491] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:44.677263 2451 raft_consensus.cc:3054] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.680528 2445 raft_consensus.cc:513] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.682760 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7340a7a916c4b2fab32e0ba2d0ce322" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:44.682899 917 raft_consensus.cc:3054] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.685725 2445 leader_election.cc:290] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:44.686012 714 leader_election.cc:343] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: b7340a7a916c4b2fab32e0ba2d0ce322
I20250112 02:52:44.686177 1177 tablet_bootstrap.cc:492] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:44.686314 1177 ts_tablet_manager.cc:1397] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.004s
I20250112 02:52:44.686909 1177 raft_consensus.cc:357] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.687100 1177 raft_consensus.cc:383] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.687177 1177 raft_consensus.cc:738] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.687388 1177 consensus_queue.cc:260] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.687940 1177 ts_tablet_manager.cc:1428] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 02:52:44.688210 1177 tablet_bootstrap.cc:492] T f69b7a0f52304ad9951a07b3ab0372e7 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:44.689000 455 catalog_manager.cc:5526] T 41d5799b5b36487da42dc43b1c70ae26 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:44.689477 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:44.690040 1177 tablet_bootstrap.cc:654] T f69b7a0f52304ad9951a07b3ab0372e7 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.691684 1181 tablet_bootstrap.cc:492] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:44.691800 1181 ts_tablet_manager.cc:1397] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.004s
I20250112 02:52:44.692449 1181 raft_consensus.cc:357] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.692601 1181 raft_consensus.cc:383] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.692279 455 catalog_manager.cc:5526] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:44.692651 1181 raft_consensus.cc:738] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.692840 1181 consensus_queue.cc:260] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.693284 1181 ts_tablet_manager.cc:1428] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 02:52:44.693528 1181 tablet_bootstrap.cc:492] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:44.695086 1181 tablet_bootstrap.cc:654] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.702474 1173 tablet_bootstrap.cc:492] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:44.702610 1173 ts_tablet_manager.cc:1397] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.073s user 0.014s sys 0.006s
I20250112 02:52:44.703249 1173 raft_consensus.cc:357] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.703485 1173 raft_consensus.cc:383] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.703550 1173 raft_consensus.cc:738] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.703734 1173 consensus_queue.cc:260] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.704144 2787 raft_consensus.cc:491] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.704191 1173 ts_tablet_manager.cc:1428] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:44.704247 2787 raft_consensus.cc:513] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.704447 1173 tablet_bootstrap.cc:492] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:44.704804 2787 leader_election.cc:290] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.705433 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb7e0181cf74bbea6036c106516a7b3" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:44.705791 1173 tablet_bootstrap.cc:654] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.706135 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb7e0181cf74bbea6036c106516a7b3" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:44.706629 854 leader_election.cc:343] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:44.707044 855 leader_election.cc:343] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.707147 855 leader_election.cc:304] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.707604 2787 raft_consensus.cc:2743] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.707639 1173 tablet_bootstrap.cc:492] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:44.707731 1173 ts_tablet_manager.cc:1397] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:44.708318 1173 raft_consensus.cc:357] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.708503 1173 raft_consensus.cc:383] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.708572 1173 raft_consensus.cc:738] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.708762 1173 consensus_queue.cc:260] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.709311 1173 ts_tablet_manager.cc:1428] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 02:52:44.710222 1173 tablet_bootstrap.cc:492] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:44.711647 1173 tablet_bootstrap.cc:654] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.716363 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 17c54c5dc86446cf85974777c8145ef6 (table default.Table_8ab566cf9c8f45eba4dc9bb85264c823 [id=a622f7a973344a388b3275adbcce6234]). Expected version 1 got 0
I20250112 02:52:44.719051 1177 tablet_bootstrap.cc:492] T f69b7a0f52304ad9951a07b3ab0372e7 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:44.719231 1177 ts_tablet_manager.cc:1397] T f69b7a0f52304ad9951a07b3ab0372e7 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.007s
I20250112 02:52:44.720045 1177 raft_consensus.cc:357] T f69b7a0f52304ad9951a07b3ab0372e7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.720315 1177 raft_consensus.cc:383] T f69b7a0f52304ad9951a07b3ab0372e7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.720412 1177 raft_consensus.cc:738] T f69b7a0f52304ad9951a07b3ab0372e7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.720671 1177 consensus_queue.cc:260] T f69b7a0f52304ad9951a07b3ab0372e7 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.721272 1177 ts_tablet_manager.cc:1428] T f69b7a0f52304ad9951a07b3ab0372e7 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:44.721609 1177 tablet_bootstrap.cc:492] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:44.721657 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:44.722546 455 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:44.729079 1173 tablet_bootstrap.cc:492] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:44.729193 1173 ts_tablet_manager.cc:1397] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20250112 02:52:44.729771 1173 raft_consensus.cc:357] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.729929 1173 raft_consensus.cc:383] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.729980 1173 raft_consensus.cc:738] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.730154 1173 consensus_queue.cc:260] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.730666 1173 ts_tablet_manager.cc:1428] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250112 02:52:44.730986 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_66ecd84e49d64f50b63b520f74b8e062 [id=3a0f5ac3329d4b60befce41c91f5ae5c] alter complete (version 1)
I20250112 02:52:44.731631 1173 tablet_bootstrap.cc:492] T ff1b3ee229c445b19405cf4a19cf8003 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:44.731819 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 3d662e0ce13f447595e5f92b94de3018
I20250112 02:52:44.732343 448 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40610 (request call id 200) took 3084 ms. Trace:
I20250112 02:52:44.732956 1173 tablet_bootstrap.cc:654] T ff1b3ee229c445b19405cf4a19cf8003 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.732424 448 rpcz_store.cc:276] 0112 02:52:41.648137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.069950 (+2421813us) service_pool.cc:225] Handling call
0112 02:52:44.070417 (+ 467us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:44.070543 (+ 126us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:52:44.070604 (+ 61us) catalog_manager.cc:2762] Locking tablets
0112 02:52:44.105117 (+ 34513us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:52:44.114008 (+ 8891us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:52:44.114082 (+ 74us) catalog_manager.cc:2810] Committing in-memory state
0112 02:52:44.731772 (+617690us) catalog_manager.cc:2814] Aborting table tasks
0112 02:52:44.732293 (+ 521us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:44.105813 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:44.105903 (+ 90us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:44.105909 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:44.105911 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:44.106013 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:44.106025 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:44.106029 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:44.106116 (+ 87us) tablet.cc:698] Row locks acquired
0112 02:52:44.106118 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:44.106143 (+ 25us) write_op.cc:270] Start()
0112 02:52:44.106165 (+ 22us) write_op.cc:276] Timestamp: P: 1736650364106055 usec, L: 0
0112 02:52:44.106169 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:44.106301 (+ 132us) log.cc:844] Serialized 756 byte log entry
0112 02:52:44.110030 (+ 3729us) op_driver.cc:464] REPLICATION: finished
0112 02:52:44.110116 (+ 86us) write_op.cc:301] APPLY: starting
0112 02:52:44.110127 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:44.110162 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:44.110191 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:44.110248 (+ 57us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:44.110274 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:44.110280 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:44.110336 (+ 56us) log.cc:844] Serialized 29 byte log entry
0112 02:52:44.110383 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.110401 (+ 18us) write_op.cc:454] Released schema lock
0112 02:52:44.110406 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:44.110463 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.110478 (+ 15us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":47,"apply.run_cpu_time_us":342,"apply.run_wall_time_us":342,"num_ops":2,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":567,"prepare.run_wall_time_us":567,"raft.queue_time_us":889,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":318,"replication_time_us":3838}]]}
I20250112 02:52:44.733371 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_3406d849cc9a4ffab1b431da38b95bcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:44.735136 2787 raft_consensus.cc:491] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.735248 2787 raft_consensus.cc:513] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.735801 1177 tablet_bootstrap.cc:654] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.735853 2787 leader_election.cc:290] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.736538 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbe30f9860e2492b90e8cedf8ff9daba" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:44.736954 854 leader_election.cc:343] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.737280 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbe30f9860e2492b90e8cedf8ff9daba" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:44.737687 855 leader_election.cc:343] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.737794 855 leader_election.cc:304] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.738114 2787 raft_consensus.cc:2743] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.738884 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d662e0ce13f447595e5f92b94de3018 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:44 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.739117 779 raft_consensus.cc:2463] T ca6f080e5fd84084b601b646d4e6c9e2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:44.739277 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d662e0ce13f447595e5f92b94de3018 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:44 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.739392 917 raft_consensus.cc:2463] T b7340a7a916c4b2fab32e0ba2d0ce322 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:44.739730 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d662e0ce13f447595e5f92b94de3018 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:44 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:44.742149 713 leader_election.cc:304] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.743530 3109 tablet_replica.cc:331] T 3d662e0ce13f447595e5f92b94de3018 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:44.743624 855 leader_election.cc:304] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.743947 2787 raft_consensus.cc:2798] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:44.743978 3111 tablet_replica.cc:331] T 3d662e0ce13f447595e5f92b94de3018 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:44.744065 2787 raft_consensus.cc:695] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:44.744125 3111 raft_consensus.cc:2238] T 3d662e0ce13f447595e5f92b94de3018 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:44.744185 3109 raft_consensus.cc:2238] T 3d662e0ce13f447595e5f92b94de3018 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:44.744240 3111 raft_consensus.cc:2267] T 3d662e0ce13f447595e5f92b94de3018 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:44.744302 3109 raft_consensus.cc:2267] T 3d662e0ce13f447595e5f92b94de3018 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:44.744330 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3406d849cc9a4ffab1b431da38b95bcf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:44.744269 2787 consensus_queue.cc:237] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.745561 3111 ts_tablet_manager.cc:1905] T 3d662e0ce13f447595e5f92b94de3018 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:44.745787 3109 ts_tablet_manager.cc:1905] T 3d662e0ce13f447595e5f92b94de3018 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:44.747967 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13] alter complete (version 1)
I20250112 02:52:44.750401 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_daa4ea5584bd498fafec7fff852d084c [id=b6d8e3bccc9c4cc694593463768a7e1c] alter complete (version 2)
I20250112 02:52:44.750211 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_4fc60c2b6a554573bcca21da7a197875 [id=b42fa8fc65e344f989a98ae411e916ed] alter complete (version 2)
I20250112 02:52:44.753528 1177 tablet_bootstrap.cc:492] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:44.753734 1177 ts_tablet_manager.cc:1397] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.000s
I20250112 02:52:44.754499 1177 raft_consensus.cc:357] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.754762 1177 raft_consensus.cc:383] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.754859 1177 raft_consensus.cc:738] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.755131 1177 consensus_queue.cc:260] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.755596 2445 raft_consensus.cc:2798] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:44.755697 2445 raft_consensus.cc:695] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:44.755934 2445 consensus_queue.cc:237] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.759763 1177 ts_tablet_manager.cc:1428] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250112 02:52:44.760408 1177 tablet_bootstrap.cc:492] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:44.762152 1177 tablet_bootstrap.cc:654] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.766878 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_1325d9a6747941458d8655d565170565 [id=85db347bdf4a4ff583062f3dddd20d8a] alter complete (version 1)
I20250112 02:52:44.767231 448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 201) took 3109 ms. Trace:
W20250112 02:52:44.769250 452 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:38361 (request call id 216) took 3091 ms (client timeout 1000 ms). Trace:
I20250112 02:52:44.771186 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet f924459d1150472aafb325a11825db23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:39 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:44.771312 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7] alter complete (version 3)
I20250112 02:52:44.771394 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet f31b777e7e0f43a28839d8c48c6a6590 with delete_type TABLET_DATA_DELETED (replaced by tablet ca2165362ee541adac724b63f39b1489) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:44.771912 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 with delete_type TABLET_DATA_DELETED (replaced by tablet 3dab0466476c4a9aa265271c1abd91a4) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:44.772289 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet b8daf4ef3593410eae87672402292be6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:44.769394 452 rpcz_store.cc:269] 0112 02:52:41.677849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.683349 (+ 5500us) service_pool.cc:225] Handling call
0112 02:52:44.769199 (+3085850us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:42.014212 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:42.014308 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:42.014315 (+ 7us) write_op.cc:435] Acquired schema lock
0112 02:52:42.014317 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:42.014526 (+ 209us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:42.014538 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:42.014543 (+ 5us) tablet.cc:682] Acquiring locks for 8 operations
0112 02:52:42.014689 (+ 146us) tablet.cc:698] Row locks acquired
0112 02:52:42.014693 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:42.014727 (+ 34us) write_op.cc:270] Start()
0112 02:52:42.014757 (+ 30us) write_op.cc:276] Timestamp: P: 1736650362014630 usec, L: 0
0112 02:52:42.014760 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:42.014951 (+ 191us) log.cc:844] Serialized 3146 byte log entry
0112 02:52:42.018845 (+ 3894us) op_driver.cc:464] REPLICATION: finished
0112 02:52:42.018956 (+ 111us) write_op.cc:301] APPLY: starting
0112 02:52:42.018972 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:42.019062 (+ 90us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:42.019092 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:42.019289 (+ 197us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:42.019329 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0112 02:52:42.019337 (+ 8us) write_op.cc:312] APPLY: finished
0112 02:52:42.019449 (+ 112us) log.cc:844] Serialized 65 byte log entry
0112 02:52:42.019549 (+ 100us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.019574 (+ 25us) write_op.cc:454] Released schema lock
0112 02:52:42.019581 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:42.019652 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.019656 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:42.028537 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:42.028643 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:42.028648 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:42.028650 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:42.028752 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:42.028762 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:42.028767 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:42.028818 (+ 51us) tablet.cc:698] Row locks acquired
0112 02:52:42.028820 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:42.028847 (+ 27us) write_op.cc:270] Start()
0112 02:52:42.028871 (+ 24us) write_op.cc:276] Timestamp: P: 1736650362028750 usec, L: 0
0112 02:52:42.028874 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:42.029008 (+ 134us) log.cc:844] Serialized 356 byte log entry
0112 02:52:42.031520 (+ 2512us) op_driver.cc:464] REPLICATION: finished
0112 02:52:42.031620 (+ 100us) write_op.cc:301] APPLY: starting
0112 02:52:42.031632 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:42.031661 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:42.031687 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:42.031726 (+ 39us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:42.031750 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:42.031757 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:42.031814 (+ 57us) log.cc:844] Serialized 23 byte log entry
0112 02:52:42.031869 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.031884 (+ 15us) write_op.cc:454] Released schema lock
0112 02:52:42.031890 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:42.031949 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.031952 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:43.721847 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:43.723180 (+ 1333us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:43.723195 (+ 15us) write_op.cc:435] Acquired schema lock
0112 02:52:43.723198 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:43.723323 (+ 125us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:43.723335 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:43.723341 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:43.723409 (+ 68us) tablet.cc:698] Row locks acquired
0112 02:52:43.723411 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:43.723456 (+ 45us) write_op.cc:270] Start()
0112 02:52:43.723846 (+ 390us) write_op.cc:276] Timestamp: P: 1736650363723303 usec, L: 0
0112 02:52:43.723851 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:43.724127 (+ 276us) log.cc:844] Serialized 356 byte log entry
0112 02:52:43.788864 (+ 64737us) op_driver.cc:464] REPLICATION: finished
0112 02:52:43.799671 (+ 10807us) write_op.cc:301] APPLY: starting
0112 02:52:43.799694 (+ 23us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:43.799727 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:43.799747 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:43.799788 (+ 41us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:43.799815 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:43.799821 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:43.799896 (+ 75us) log.cc:844] Serialized 23 byte log entry
0112 02:52:43.800748 (+ 852us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:43.800762 (+ 14us) write_op.cc:454] Released schema lock
0112 02:52:43.800768 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":65,"apply.run_cpu_time_us":687,"apply.run_wall_time_us":687,"num_ops":8,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":840,"prepare.run_wall_time_us":840,"raft.queue_time_us":78,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":238,"replication_time_us":4059}],["op",{"apply.queue_time_us":58,"apply.run_cpu_time_us":322,"apply.run_wall_time_us":321,"num_ops":1,"prepare.queue_time_us":78,"prepare.run_cpu_time_us":558,"prepare.run_wall_time_us":557,"raft.queue_time_us":52,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":182,"replication_time_us":2622}],["op",{"apply.queue_time_us":10766,"apply.run_cpu_time_us":356,"apply.run_wall_time_us":1150,"num_ops":1,"prepare.queue_time_us":474640,"prepare.run_cpu_time_us":671,"prepare.run_wall_time_us":2345,"replication_time_us":64986,"thread_start_us":29430,"threads_started":1}]]}
I20250112 02:52:44.773036 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_8684181f6c4b40eb8948c3687408a1c4 [id=78ee5f3a07714849a10192e80c425264] alter complete (version 1)
I20250112 02:52:44.767329 448 rpcz_store.cc:276] 0112 02:52:41.657349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.733143 (+3075794us) service_pool.cc:225] Handling call
0112 02:52:44.744427 (+ 11284us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:44.744964 (+ 537us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:44.766903 (+ 21939us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:44.767089 (+ 186us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:44.767171 (+ 82us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:44.745587 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:44.745703 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:44.745713 (+ 10us) write_op.cc:435] Acquired schema lock
0112 02:52:44.745717 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:44.745844 (+ 127us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:44.745861 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:44.745869 (+ 8us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:44.745966 (+ 97us) tablet.cc:698] Row locks acquired
0112 02:52:44.745970 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:44.746008 (+ 38us) write_op.cc:270] Start()
0112 02:52:44.746046 (+ 38us) write_op.cc:276] Timestamp: P: 1736650364745890 usec, L: 0
0112 02:52:44.746052 (+ 6us) op_driver.cc:348] REPLICATION: starting
0112 02:52:44.746444 (+ 392us) log.cc:844] Serialized 433 byte log entry
0112 02:52:44.751580 (+ 5136us) op_driver.cc:464] REPLICATION: finished
0112 02:52:44.751683 (+ 103us) write_op.cc:301] APPLY: starting
0112 02:52:44.751695 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:44.751731 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:44.751760 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:44.751878 (+ 118us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:44.751914 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:44.751922 (+ 8us) write_op.cc:312] APPLY: finished
0112 02:52:44.751988 (+ 66us) log.cc:844] Serialized 29 byte log entry
0112 02:52:44.752041 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.752056 (+ 15us) write_op.cc:454] Released schema lock
0112 02:52:44.752062 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:44.752125 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.752128 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":52,"apply.run_cpu_time_us":435,"apply.run_wall_time_us":435,"num_ops":2,"prepare.queue_time_us":91,"prepare.run_cpu_time_us":998,"prepare.run_wall_time_us":999,"raft.queue_time_us":1162,"raft.run_cpu_time_us":386,"raft.run_wall_time_us":387,"replication_time_us":5501}]]}
I20250112 02:52:44.773758 826 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42341
I20250112 02:52:44.781888 2451 raft_consensus.cc:513] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.782833 826 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:44.783192 2451 leader_election.cc:290] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:44.784088 714 leader_election.cc:343] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 5804fc4130f84c69832c1022b7bd7fa5
I20250112 02:52:44.784713 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5804fc4130f84c69832c1022b7bd7fa5" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:44.784941 917 raft_consensus.cc:3054] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.790925 3110 tablet_replica.cc:331] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:44.791481 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_54e45ab6e6d14005bfe1cc06501f6d9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:44.791486 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_31ea460b9f1d4424932127e3eb550a27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:44.792034 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54e45ab6e6d14005bfe1cc06501f6d9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:44.792034 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31ea460b9f1d4424932127e3eb550a27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:44.793430 3110 raft_consensus.cc:2238] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:44.793792 3110 raft_consensus.cc:2267] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:44.794565 3110 ts_tablet_manager.cc:1905] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:44.796511 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_bfb5e14e3f334af2a4459f8f3cef66a8 [id=d8a4cc77c79449da82b721854466fc55] alter complete (version 1)
I20250112 02:52:44.797013 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_c16c321e8c3741838921ff50f04b8261 [id=78dff38cafdc48ddbc46b91a548c2022] alter complete (version 1)
I20250112 02:52:44.804301 760 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ffdc237f8640f0b378809f93fc6292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.805131 756 tablet_service.cc:1514] Processing DeleteTablet for tablet b124c15736084e6d8f69b4fb7c41cf39 with delete_type TABLET_DATA_DELETED (replaced by tablet 97c7ce72d02049089c653684c53cb53c) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.805353 744 tablet_service.cc:1514] Processing DeleteTablet for tablet c29a9ca43e034b43a00207e5847bbee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.805500 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f4bd2df0ce45dabc55373f1154d3ac with delete_type TABLET_DATA_DELETED (replaced by tablet f559dedfb7854cf890bef362c50b4213) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.805552 754 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5b3c712c0f448ea9b14ab93dd731ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.805507 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1954e7859f422f9690fe4a9384a8bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.805928 749 tablet_service.cc:1514] Processing DeleteTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.806066 747 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e7882e07784bdda18dc1ffd20963c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.806279 760 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d411f906b4d4eb68343bdcc493e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.809798 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 8429a3d8636d4e2ca4c1191f85b9266b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.816625 897 tablet_service.cc:1514] Processing DeleteTablet for tablet b124c15736084e6d8f69b4fb7c41cf39 with delete_type TABLET_DATA_DELETED (replaced by tablet 97c7ce72d02049089c653684c53cb53c) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.816692 890 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1b3ee229c445b19405cf4a19cf8003 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.816900 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e7a2e88c374e949ced92c557225cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.817055 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b9280db29e54d41af1bed2fa5aa5535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.817435 756 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1b3ee229c445b19405cf4a19cf8003 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.817636 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e7a2e88c374e949ced92c557225cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:44.818403 451 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 3dab0466476c4a9aa265271c1abd91a4 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 3dab0466476c4a9aa265271c1abd91a4 consensus state has no leader
I20250112 02:52:44.818419 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b9280db29e54d41af1bed2fa5aa5535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.818425 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c04f1a2cb04950a9f6170e252c19b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.818852 1181 tablet_bootstrap.cc:492] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:44.818895 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 781a9943f705477cb6eae16df4468d2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.819018 1181 ts_tablet_manager.cc:1397] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.126s user 0.009s sys 0.019s
I20250112 02:52:44.819494 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b351c563d749df806ab5d0331a4ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.820207 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 681a930541a84f0197ecafe670278e70 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b1d46c5395d4defb2d0e2948807eaf4) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.820406 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 13578542e9e741e8817e0ee63b10f1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.820883 747 tablet_service.cc:1514] Processing DeleteTablet for tablet cbe30f9860e2492b90e8cedf8ff9daba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.821022 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c821ea3ca4946c290c17135366e0cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.821067 2787 raft_consensus.cc:491] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.821211 2787 raft_consensus.cc:513] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.821815 2787 leader_election.cc:290] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.822556 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f52b838214c54ac69ad30dac6e56669a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:44.822940 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f52b838214c54ac69ad30dac6e56669a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:44.823523 452 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 202) took 3133 ms. Trace:
I20250112 02:52:44.823642 452 rpcz_store.cc:276] 0112 02:52:41.690144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.773302 (+3083158us) service_pool.cc:225] Handling call
0112 02:52:44.791394 (+ 18092us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd72d03 00007f3e13e75236 00007f3e13e74f65 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:44.792411 (+ 1017us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:44.792954 (+ 543us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:44.823263 (+ 30309us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:44.823382 (+ 119us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:44.823460 (+ 78us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:44.793405 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:44.793491 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:44.793496 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:44.793498 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:44.793589 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:44.793597 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:44.793601 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:44.793664 (+ 63us) tablet.cc:698] Row locks acquired
0112 02:52:44.793667 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:44.793690 (+ 23us) write_op.cc:270] Start()
0112 02:52:44.793711 (+ 21us) write_op.cc:276] Timestamp: P: 1736650364793669 usec, L: 0
0112 02:52:44.793714 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:44.793832 (+ 118us) log.cc:844] Serialized 433 byte log entry
0112 02:52:44.821794 (+ 27962us) op_driver.cc:464] REPLICATION: finished
0112 02:52:44.822515 (+ 721us) write_op.cc:301] APPLY: starting
0112 02:52:44.822525 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:44.822552 (+ 27us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:44.822581 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:44.822671 (+ 90us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:44.822699 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:44.822706 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:44.822754 (+ 48us) log.cc:844] Serialized 29 byte log entry
0112 02:52:44.822788 (+ 34us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.822799 (+ 11us) write_op.cc:454] Released schema lock
0112 02:52:44.822804 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:44.822856 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.822860 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":40659456,"child_traces":[["op",{"apply.queue_time_us":678,"apply.run_cpu_time_us":336,"apply.run_wall_time_us":336,"num_ops":2,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":494,"prepare.run_wall_time_us":494,"replication_time_us":28064}]]}
I20250112 02:52:44.824715 448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 203) took 3003 ms. Trace:
I20250112 02:52:44.824798 448 rpcz_store.cc:276] 0112 02:52:41.820993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.774826 (+2953833us) service_pool.cc:225] Handling call
0112 02:52:44.791351 (+ 16525us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd72d03 00007f3e13e75236 00007f3e13e74f65 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:44.792155 (+ 804us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:44.795626 (+ 3471us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:44.824483 (+ 28857us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:44.824578 (+ 95us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:44.824658 (+ 80us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:44.795976 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:44.796045 (+ 69us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:44.796049 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:44.796051 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:44.796124 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:44.796130 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:44.796134 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:44.796187 (+ 53us) tablet.cc:698] Row locks acquired
0112 02:52:44.796188 (+ 1us) write_op.cc:260] PREPARE: finished
0112 02:52:44.796213 (+ 25us) write_op.cc:270] Start()
0112 02:52:44.796234 (+ 21us) write_op.cc:276] Timestamp: P: 1736650364796191 usec, L: 0
0112 02:52:44.796237 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:44.796353 (+ 116us) log.cc:844] Serialized 433 byte log entry
0112 02:52:44.821845 (+ 25492us) op_driver.cc:464] REPLICATION: finished
0112 02:52:44.822912 (+ 1067us) write_op.cc:301] APPLY: starting
0112 02:52:44.822920 (+ 8us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:44.822949 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:44.822974 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:44.823048 (+ 74us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:44.823070 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:44.823077 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:44.823116 (+ 39us) log.cc:844] Serialized 29 byte log entry
0112 02:52:44.823146 (+ 30us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.823156 (+ 10us) write_op.cc:454] Released schema lock
0112 02:52:44.823161 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":37365760,"child_traces":[["op",{"apply.queue_time_us":1043,"apply.run_cpu_time_us":293,"apply.run_wall_time_us":292,"num_ops":2,"prepare.queue_time_us":38,"prepare.run_cpu_time_us":425,"prepare.run_wall_time_us":424,"replication_time_us":25597,"threads_started":1}]]}
I20250112 02:52:44.825381 2451 raft_consensus.cc:491] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.825510 2451 raft_consensus.cc:513] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.826611 2451 leader_election.cc:290] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:44.826843 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c04f1a2cb04950a9f6170e252c19b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.827039 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 795ac7eaf81d48bb926865d9ebb388b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 818cd6362b94473ea06ed68fa5ffd1bc) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.827221 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 781a9943f705477cb6eae16df4468d2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.827397 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 76009bac22ce4136b148b493a004a0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.827435 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd81852962fb43449e35ef45dc1daaad" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:44.827591 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a88463991294bd8a335337f8b072c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.827653 910 raft_consensus.cc:2463] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:44.828327 713 leader_election.cc:304] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
W20250112 02:52:44.828630 854 leader_election.cc:343] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.827188 897 tablet_service.cc:1514] Processing DeleteTablet for tablet cbe30f9860e2492b90e8cedf8ff9daba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.829010 3106 raft_consensus.cc:2798] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:44.829092 3106 raft_consensus.cc:491] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:44.829146 3106 raft_consensus.cc:3054] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.827605 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 8429a3d8636d4e2ca4c1191f85b9266b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:38786
W20250112 02:52:44.829941 855 leader_election.cc:343] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.830044 855 leader_election.cc:304] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.820415 756 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cb54003a83472cb9c57ace3e65b0be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.831218 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 795ac7eaf81d48bb926865d9ebb388b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 818cd6362b94473ea06ed68fa5ffd1bc) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.831377 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 54215c55966a46e1ab96e352c4e3764e with delete_type TABLET_DATA_DELETED (replaced by tablet a470f84d50df4cf0b4a1e5d248b4c32a) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.831553 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a88463991294bd8a335337f8b072c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.832026 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd81852962fb43449e35ef45dc1daaad" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:44.832190 1052 raft_consensus.cc:2463] T fd81852962fb43449e35ef45dc1daaad P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:44.832440 1173 tablet_bootstrap.cc:492] T ff1b3ee229c445b19405cf4a19cf8003 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:44.832554 1173 ts_tablet_manager.cc:1397] T ff1b3ee229c445b19405cf4a19cf8003 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.101s user 0.006s sys 0.002s
I20250112 02:52:44.832490 2179 consensus_queue.cc:1035] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:44.832592 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 058a9ca089cd4967887b0a1127c7d03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.833176 2673 raft_consensus.cc:2743] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.833338 1173 raft_consensus.cc:357] T ff1b3ee229c445b19405cf4a19cf8003 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.833496 1173 raft_consensus.cc:383] T ff1b3ee229c445b19405cf4a19cf8003 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.833544 1173 raft_consensus.cc:738] T ff1b3ee229c445b19405cf4a19cf8003 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.833722 1173 consensus_queue.cc:260] T ff1b3ee229c445b19405cf4a19cf8003 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.834599 2228 consensus_queue.cc:1035] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:44.835072 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 76009bac22ce4136b148b493a004a0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.836506 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 681a930541a84f0197ecafe670278e70 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b1d46c5395d4defb2d0e2948807eaf4) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.837385 1173 ts_tablet_manager.cc:1428] T ff1b3ee229c445b19405cf4a19cf8003 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 02:52:44.837783 1173 tablet_bootstrap.cc:492] T 2d3d291ca3494f05ac23c7fe293e0bd9 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:44.838467 1177 tablet_bootstrap.cc:492] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:44.838711 1177 ts_tablet_manager.cc:1397] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.078s user 0.002s sys 0.002s
I20250112 02:52:44.839854 1177 raft_consensus.cc:357] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.840137 1173 tablet_bootstrap.cc:654] T 2d3d291ca3494f05ac23c7fe293e0bd9 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.840155 1177 raft_consensus.cc:383] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.840350 1177 raft_consensus.cc:738] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.840695 1177 consensus_queue.cc:260] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.841498 1177 ts_tablet_manager.cc:1428] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:44.842514 1181 raft_consensus.cc:357] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.843019 1177 tablet_bootstrap.cc:492] T 9bbef42e11a843b18ed582ab4d04a92a P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:44.843441 1181 raft_consensus.cc:383] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.843691 1181 raft_consensus.cc:738] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.844115 1181 consensus_queue.cc:260] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.844415 2977 consensus_queue.cc:1035] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:44.845036 1181 ts_tablet_manager.cc:1428] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.026s user 0.001s sys 0.000s
I20250112 02:52:44.845650 1181 tablet_bootstrap.cc:492] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:44.846169 1177 tablet_bootstrap.cc:654] T 9bbef42e11a843b18ed582ab4d04a92a P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:44.853315 989 consensus_peers.cc:487] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82b351c563d749df806ab5d0331a4ef2. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:44.857467 890 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cb54003a83472cb9c57ace3e65b0be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.862129 1181 tablet_bootstrap.cc:654] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:44.865136 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 3dab0466476c4a9aa265271c1abd91a4 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3dab0466476c4a9aa265271c1abd91a4 consensus state has no leader
I20250112 02:52:44.868942 890 tablet_service.cc:1514] Processing DeleteTablet for tablet f3771166b77e4f90aaa116ab1a9dc299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:39 UTC) from {username='slave'} at 127.0.0.1:38786
W20250112 02:52:44.870270 449 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:58959 (request call id 243) took 3221 ms (client timeout 1000 ms). Trace:
I20250112 02:52:44.873191 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 with delete_type TABLET_DATA_DELETED (replaced by tablet 3dab0466476c4a9aa265271c1abd91a4) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.873472 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 058a9ca089cd4967887b0a1127c7d03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.873692 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 940affdc2c484b7d9fd2de17f9d1cea7 with delete_type TABLET_DATA_DELETED (replaced by tablet ab183c43f6a64270ad744e61364f0a99) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.873888 898 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e7882e07784bdda18dc1ffd20963c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.873476 897 tablet_service.cc:1514] Processing DeleteTablet for tablet c29a9ca43e034b43a00207e5847bbee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.874542 756 tablet_service.cc:1514] Processing DeleteTablet for tablet f3771166b77e4f90aaa116ab1a9dc299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:39 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.878151 3109 ts_tablet_manager.cc:1918] T 3d662e0ce13f447595e5f92b94de3018 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:44.878258 3109 log.cc:1198] T 3d662e0ce13f447595e5f92b94de3018 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/3d662e0ce13f447595e5f92b94de3018
I20250112 02:52:44.878618 3109 ts_tablet_manager.cc:1939] T 3d662e0ce13f447595e5f92b94de3018 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:44.881572 3111 ts_tablet_manager.cc:1918] T 3d662e0ce13f447595e5f92b94de3018 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:44.881723 3111 log.cc:1198] T 3d662e0ce13f447595e5f92b94de3018 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/3d662e0ce13f447595e5f92b94de3018
I20250112 02:52:44.882100 3111 ts_tablet_manager.cc:1939] T 3d662e0ce13f447595e5f92b94de3018 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:44.882313 917 raft_consensus.cc:2463] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:44.882911 713 leader_election.cc:304] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.884549 3109 tablet_replica.cc:331] T a3ffdc237f8640f0b378809f93fc6292 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:44.884824 3109 raft_consensus.cc:2238] T a3ffdc237f8640f0b378809f93fc6292 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:44.884979 3109 raft_consensus.cc:2267] T a3ffdc237f8640f0b378809f93fc6292 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:44.885555 3109 ts_tablet_manager.cc:1905] T a3ffdc237f8640f0b378809f93fc6292 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:44.890249 3106 raft_consensus.cc:513] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.890954 3106 leader_election.cc:290] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:44.891130 3106 raft_consensus.cc:2798] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:44.891690 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd81852962fb43449e35ef45dc1daaad" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:44.891855 1050 raft_consensus.cc:3054] T fd81852962fb43449e35ef45dc1daaad P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.894801 3106 raft_consensus.cc:695] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:44.898604 1052 raft_consensus.cc:3054] T 5a3af0ea4e954053ae89814f3e74e6b7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.896193 3106 consensus_queue.cc:237] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.900633 1181 tablet_bootstrap.cc:492] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:44.900781 1181 ts_tablet_manager.cc:1397] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.000s
I20250112 02:52:44.901659 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]) successfully deleted
I20250112 02:52:44.907593 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 940affdc2c484b7d9fd2de17f9d1cea7 with delete_type TABLET_DATA_DELETED (replaced by tablet ab183c43f6a64270ad744e61364f0a99) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.908061 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:44.908434 1181 raft_consensus.cc:357] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.908710 1181 raft_consensus.cc:383] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.908808 1181 raft_consensus.cc:738] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.909271 756 tablet_service.cc:1514] Processing DeleteTablet for tablet bab4978edea0445fb0456e88e7fb4c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:44.909320 451 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:37987 (request call id 257) took 3263 ms (client timeout 1000 ms). Trace:
I20250112 02:52:44.909457 747 tablet_service.cc:1514] Processing DeleteTablet for tablet b1d9fae290ec4281839991f9d40b105c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:44.909534 451 rpcz_store.cc:269] 0112 02:52:41.646048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.663643 (+ 17595us) service_pool.cc:225] Handling call
0112 02:52:41.774554 (+110911us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd6d6b8 00007f3e13e72716 00007f3e13e72445 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:44.909282 (+3134728us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35962,"reported_tablets":181,"spinlock_wait_cycles":235091328}
W20250112 02:52:44.910079 989 consensus_peers.cc:487] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f776e2a6ec704cf9b64483bc7d5b1dff. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:44.910172 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 93aba9b4d1dd485cb62c4827c916a1d5 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:44.910084 1181 consensus_queue.cc:260] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.911406 1181 ts_tablet_manager.cc:1428] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250112 02:52:44.911605 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:44.911854 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:44.912895 1181 tablet_bootstrap.cc:492] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:44.913522 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd81852962fb43449e35ef45dc1daaad" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:44.914908 965 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42341
I20250112 02:52:44.914770 452 catalog_manager.cc:5526] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:44.915042 965 heartbeater.cc:502] Master 127.0.104.62:42341 was elected leader, sending a full tablet report...
I20250112 02:52:44.915192 910 raft_consensus.cc:3054] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.915786 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:44.918741 1173 tablet_bootstrap.cc:492] T 2d3d291ca3494f05ac23c7fe293e0bd9 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:44.918861 1173 ts_tablet_manager.cc:1397] T 2d3d291ca3494f05ac23c7fe293e0bd9 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.081s user 0.006s sys 0.006s
I20250112 02:52:44.919560 1173 raft_consensus.cc:357] T 2d3d291ca3494f05ac23c7fe293e0bd9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.919744 1173 raft_consensus.cc:383] T 2d3d291ca3494f05ac23c7fe293e0bd9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.919786 1173 raft_consensus.cc:738] T 2d3d291ca3494f05ac23c7fe293e0bd9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.919929 1173 consensus_queue.cc:260] T 2d3d291ca3494f05ac23c7fe293e0bd9 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.920336 1173 ts_tablet_manager.cc:1428] T 2d3d291ca3494f05ac23c7fe293e0bd9 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 02:52:44.921211 1173 tablet_bootstrap.cc:492] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:44.922686 1173 tablet_bootstrap.cc:654] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.914659 1181 tablet_bootstrap.cc:654] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.923842 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 93aba9b4d1dd485cb62c4827c916a1d5
I20250112 02:52:44.924173 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:44.924350 451 catalog_manager.cc:6596] ReplaceTablet: tablet 93aba9b4d1dd485cb62c4827c916a1d5 deleted and replaced by tablet e3454657cbd74544b0ae962736e4b7b5
I20250112 02:52:44.924706 451 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:40610 (request call id 204) took 3095 ms. Trace:
I20250112 02:52:44.924923 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 93aba9b4d1dd485cb62c4827c916a1d5 with delete_type TABLET_DATA_DELETED (replaced by tablet e3454657cbd74544b0ae962736e4b7b5) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:44.925136 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 93aba9b4d1dd485cb62c4827c916a1d5 with delete_type TABLET_DATA_DELETED (replaced by tablet e3454657cbd74544b0ae962736e4b7b5) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.924799 451 rpcz_store.cc:276] 0112 02:52:41.828564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.909880 (+3081316us) service_pool.cc:225] Handling call
0112 02:52:44.924424 (+ 14544us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:44.911855 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:44.911960 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:44.911966 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:44.911968 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:44.912082 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:44.912095 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:44.912100 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:44.912174 (+ 74us) tablet.cc:698] Row locks acquired
0112 02:52:44.912176 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:44.912208 (+ 32us) write_op.cc:270] Start()
0112 02:52:44.912237 (+ 29us) write_op.cc:276] Timestamp: P: 1736650364912159 usec, L: 0
0112 02:52:44.912241 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:44.912405 (+ 164us) log.cc:844] Serialized 622 byte log entry
0112 02:52:44.915309 (+ 2904us) op_driver.cc:464] REPLICATION: finished
0112 02:52:44.915726 (+ 417us) write_op.cc:301] APPLY: starting
0112 02:52:44.915740 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:44.915781 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:44.915810 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:44.915898 (+ 88us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:44.915947 (+ 49us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:44.915953 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:44.916028 (+ 75us) log.cc:844] Serialized 29 byte log entry
0112 02:52:44.916231 (+ 203us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.916247 (+ 16us) write_op.cc:454] Released schema lock
0112 02:52:44.916253 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:44.916309 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.916312 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":356,"apply.run_cpu_time_us":431,"apply.run_wall_time_us":580,"num_ops":2,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":663,"prepare.run_wall_time_us":663,"raft.queue_time_us":39,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":588,"replication_time_us":3044}]]}
I20250112 02:52:44.925185 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 93aba9b4d1dd485cb62c4827c916a1d5 with delete_type TABLET_DATA_DELETED (replaced by tablet e3454657cbd74544b0ae962736e4b7b5) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:44.925787 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_294f7525beaa446ba48d602cc44838ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:44.910035 449 rpcz_store.cc:269] 0112 02:52:41.648361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:41.668382 (+ 20021us) service_pool.cc:225] Handling call
0112 02:52:44.870225 (+3201843us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:42.263452 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:42.263576 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:42.263582 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:42.263584 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:42.263683 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:42.263696 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:42.263700 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:42.263754 (+ 54us) tablet.cc:698] Row locks acquired
0112 02:52:42.263755 (+ 1us) write_op.cc:260] PREPARE: finished
0112 02:52:42.263787 (+ 32us) write_op.cc:270] Start()
0112 02:52:42.263814 (+ 27us) write_op.cc:276] Timestamp: P: 1736650362263757 usec, L: 0
0112 02:52:42.263818 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:42.263987 (+ 169us) log.cc:844] Serialized 527 byte log entry
0112 02:52:42.266560 (+ 2573us) op_driver.cc:464] REPLICATION: finished
0112 02:52:42.266672 (+ 112us) write_op.cc:301] APPLY: starting
0112 02:52:42.266684 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:42.266715 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:42.266737 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:42.266776 (+ 39us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:42.266803 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:42.266809 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:42.266883 (+ 74us) log.cc:844] Serialized 23 byte log entry
0112 02:52:42.266935 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.266951 (+ 16us) write_op.cc:454] Released schema lock
0112 02:52:42.266958 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:42.267030 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:42.267036 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":14797,"reported_tablets":170,"child_traces":[["op",{"apply.queue_time_us":67,"apply.run_cpu_time_us":348,"apply.run_wall_time_us":348,"num_ops":1,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":642,"prepare.run_wall_time_us":642,"raft.queue_time_us":100,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":265,"replication_time_us":2718,"spinlock_wait_cycles":110208}]]}
W20250112 02:52:44.927090 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_294f7525beaa446ba48d602cc44838ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:44.927263 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:44.930939 2977 raft_consensus.cc:491] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.931056 2977 raft_consensus.cc:513] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.931586 2977 leader_election.cc:290] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:44.932684 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbef42e11a843b18ed582ab4d04a92a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:44.933295 991 leader_election.cc:343] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.934178 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_9f9aeec4d85146b5bbe19873ed71d792"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:44.934496 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbef42e11a843b18ed582ab4d04a92a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:44.934666 917 raft_consensus.cc:2463] T 9bbef42e11a843b18ed582ab4d04a92a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
W20250112 02:52:44.935055 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f9aeec4d85146b5bbe19873ed71d792 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:44.935111 989 leader_election.cc:304] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:44.935518 2977 raft_consensus.cc:2798] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:44.935590 2977 raft_consensus.cc:491] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:44.935650 2977 raft_consensus.cc:3054] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.937517 1181 tablet_bootstrap.cc:492] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:44.937633 1181 ts_tablet_manager.cc:1397] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.002s
I20250112 02:52:44.938167 1181 raft_consensus.cc:357] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.938319 1181 raft_consensus.cc:383] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.938365 1181 raft_consensus.cc:738] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.938546 1181 consensus_queue.cc:260] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.938959 1181 ts_tablet_manager.cc:1428] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 02:52:44.939167 1181 tablet_bootstrap.cc:492] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:44.940340 1181 tablet_bootstrap.cc:654] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:44.941752 2787 raft_consensus.cc:491] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.941854 2787 raft_consensus.cc:513] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.942415 2787 leader_election.cc:290] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.942625 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_f952a0479e8d478bbd9484fc5bf041c8 [id=c01b653cd9fe45b2af721eb97936c5d4] alter complete (version 1)
I20250112 02:52:44.944281 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3a4e76b724452c8b238aae0198e297" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:44.945269 989 consensus_peers.cc:487] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3ffdc237f8640f0b378809f93fc6292. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:52:44.945454 854 leader_election.cc:343] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.945744 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3a4e76b724452c8b238aae0198e297" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:44.946219 855 leader_election.cc:343] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.946313 855 leader_election.cc:304] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:44.946609 2787 raft_consensus.cc:2743] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:44.962085 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_e404faebac7b4fa79ca69a843876490d [id=adfb79cc90204828b62501c5c395346a] alter complete (version 1)
I20250112 02:52:44.961241 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]) successfully deleted
I20250112 02:52:44.964417 774 raft_consensus.cc:3054] T 9b81aee21510452881948e41d7aba9be P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:44.964565 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f924459d1150472aafb325a11825db23 because the tablet was not found. No further retry: Not found: Tablet not found: f924459d1150472aafb325a11825db23
W20250112 02:52:44.965742 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f31b777e7e0f43a28839d8c48c6a6590 because the tablet was not found. No further retry: Not found: Tablet not found: f31b777e7e0f43a28839d8c48c6a6590
W20250112 02:52:44.965984 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 because the tablet was not found. No further retry: Not found: Tablet not found: 3e378a6a9b4a455c9f5e1d1ec0fd6ca0
W20250112 02:52:44.966711 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b8daf4ef3593410eae87672402292be6 because the tablet was not found. No further retry: Not found: Tablet not found: b8daf4ef3593410eae87672402292be6
W20250112 02:52:44.966981 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 93aba9b4d1dd485cb62c4827c916a1d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93aba9b4d1dd485cb62c4827c916a1d5 already in progress: creating tablet
I20250112 02:52:44.967406 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_b8e142e93c8d481798ca4063a3d26e82 [id=485dd281891f45e8b407b306572d6e29] alter complete (version 3)
W20250112 02:52:44.970593 447 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:38361 (request call id 217) took 2221 ms (client timeout 1000 ms). Trace:
W20250112 02:52:44.971289 447 rpcz_store.cc:269] 0112 02:52:42.749402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:43.246721 (+497319us) service_pool.cc:225] Handling call
0112 02:52:44.970547 (+1723826us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:43.795172 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:43.795275 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:43.795281 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:43.795283 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:43.795417 (+ 134us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:43.795426 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:43.795430 (+ 4us) tablet.cc:682] Acquiring locks for 3 operations
0112 02:52:43.795566 (+ 136us) tablet.cc:698] Row locks acquired
0112 02:52:43.795570 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:43.795601 (+ 31us) write_op.cc:270] Start()
0112 02:52:43.795630 (+ 29us) write_op.cc:276] Timestamp: P: 1736650363795572 usec, L: 0
0112 02:52:43.795634 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:43.818691 (+ 23057us) log.cc:844] Serialized 1275 byte log entry
0112 02:52:43.819206 (+ 515us) spinlock_profiling.cc:243] Waited 22.9 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 000000000096ceda 00007f3e150e3632 00007f3e1646beb5 00007f3e1646a9d9 00007f3e16472e00 00007f3e16472c0d 0000000000a71ecc 00007f3e098b0b64 00007f3e098b418b 00007f3e098b4008 0000000000a71ecc
0112 02:52:43.860282 (+ 41076us) op_driver.cc:464] REPLICATION: finished
0112 02:52:43.860385 (+ 103us) write_op.cc:301] APPLY: starting
0112 02:52:43.860395 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:43.860471 (+ 76us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:43.860474 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:43.860528 (+ 54us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:43.860550 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0112 02:52:43.860557 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:43.860620 (+ 63us) log.cc:844] Serialized 35 byte log entry
0112 02:52:43.860668 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:43.860685 (+ 17us) write_op.cc:454] Released schema lock
0112 02:52:43.860690 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:43.860741 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:43.860744 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:43.867816 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:43.867901 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:43.867905 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:43.867908 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:43.867980 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:43.867989 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:43.867993 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:43.868038 (+ 45us) tablet.cc:698] Row locks acquired
0112 02:52:43.868040 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:43.868084 (+ 44us) write_op.cc:270] Start()
0112 02:52:43.868105 (+ 21us) write_op.cc:276] Timestamp: P: 1736650363868043 usec, L: 0
0112 02:52:43.868109 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:43.868277 (+ 168us) log.cc:844] Serialized 356 byte log entry
0112 02:52:43.875741 (+ 7464us) op_driver.cc:464] REPLICATION: finished
0112 02:52:43.875860 (+ 119us) write_op.cc:301] APPLY: starting
0112 02:52:43.875873 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:43.875908 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:43.875943 (+ 35us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:43.875988 (+ 45us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:43.876019 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:43.876028 (+ 9us) write_op.cc:312] APPLY: finished
0112 02:52:43.876087 (+ 59us) log.cc:844] Serialized 23 byte log entry
0112 02:52:43.876134 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:43.876151 (+ 17us) write_op.cc:454] Released schema lock
0112 02:52:43.876158 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:43.876236 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:43.876241 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:44.749337 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:44.749414 (+ 77us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:44.749419 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:44.749422 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:44.749564 (+ 142us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:44.749571 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:44.749575 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:44.749621 (+ 46us) tablet.cc:698] Row locks acquired
0112 02:52:44.749623 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:44.749652 (+ 29us) write_op.cc:270] Start()
0112 02:52:44.749671 (+ 19us) write_op.cc:276] Timestamp: P: 1736650364749537 usec, L: 0
0112 02:52:44.749674 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:44.749807 (+ 133us) log.cc:844] Serialized 356 byte log entry
0112 02:52:44.764836 (+ 15029us) op_driver.cc:464] REPLICATION: finished
0112 02:52:44.765296 (+ 460us) write_op.cc:301] APPLY: starting
0112 02:52:44.765312 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:44.765347 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:44.765372 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:44.765420 (+ 48us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:44.765455 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:44.765465 (+ 10us) write_op.cc:312] APPLY: finished
0112 02:52:44.765529 (+ 64us) log.cc:844] Serialized 23 byte log entry
0112 02:52:44.765574 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.765600 (+ 26us) write_op.cc:454] Released schema lock
0112 02:52:44.765607 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:44.765680 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.765685 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:44.768567 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:44.768654 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:44.768659 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:44.768662 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:44.768743 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:44.768755 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:44.768760 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:44.768805 (+ 45us) tablet.cc:698] Row locks acquired
0112 02:52:44.768807 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:44.768832 (+ 25us) write_op.cc:270] Start()
0112 02:52:44.768854 (+ 22us) write_op.cc:276] Timestamp: P: 1736650364768814 usec, L: 0
0112 02:52:44.768858 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:44.768972 (+ 114us) log.cc:844] Serialized 356 byte log entry
0112 02:52:44.796004 (+ 27032us) op_driver.cc:464] REPLICATION: finished
0112 02:52:44.796098 (+ 94us) write_op.cc:301] APPLY: starting
0112 02:52:44.796105 (+ 7us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:44.796124 (+ 19us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:44.796166 (+ 42us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:44.796213 (+ 47us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:44.796239 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:44.796246 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:44.796304 (+ 58us) log.cc:844] Serialized 23 byte log entry
0112 02:52:44.796364 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.796371 (+ 7us) write_op.cc:454] Released schema lock
0112 02:52:44.796374 (+ 3us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:44.796425 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.796428 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:44.946480 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:44.946547 (+ 67us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:44.946551 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:44.946553 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:44.946642 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:44.946650 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:44.946654 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:44.946716 (+ 62us) tablet.cc:698] Row locks acquired
0112 02:52:44.946718 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:44.946744 (+ 26us) write_op.cc:270] Start()
0112 02:52:44.946769 (+ 25us) write_op.cc:276] Timestamp: P: 1736650364946690 usec, L: 0
0112 02:52:44.946772 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:44.946902 (+ 130us) log.cc:844] Serialized 356 byte log entry
0112 02:52:44.956736 (+ 9834us) op_driver.cc:464] REPLICATION: finished
0112 02:52:44.960789 (+ 4053us) write_op.cc:301] APPLY: starting
0112 02:52:44.960798 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:44.960838 (+ 40us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:44.960856 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:44.960889 (+ 33us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:44.960911 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:44.960918 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:44.961444 (+ 526us) log.cc:844] Serialized 23 byte log entry
0112 02:52:44.961489 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.961500 (+ 11us) write_op.cc:454] Released schema lock
0112 02:52:44.961506 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:44.961578 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.961583 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":17318,"reported_tablets":35,"spinlock_wait_cycles":768,"child_traces":[["op",{"apply.queue_time_us":55,"apply.run_cpu_time_us":348,"apply.run_wall_time_us":347,"num_ops":3,"prepare.queue_time_us":1834,"prepare.run_cpu_time_us":1235,"prepare.run_wall_time_us":24067,"replication_time_us":64626,"spinlock_wait_cycles":52644608}],["op",{"apply.queue_time_us":74,"apply.run_cpu_time_us":368,"apply.run_wall_time_us":368,"num_ops":1,"prepare.queue_time_us":5374,"prepare.run_cpu_time_us":524,"prepare.run_wall_time_us":525,"replication_time_us":7603}],["op",{"apply.queue_time_us":389,"apply.run_cpu_time_us":377,"apply.run_wall_time_us":378,"num_ops":1,"prepare.queue_time_us":528,"prepare.run_cpu_time_us":522,"prepare.run_wall_time_us":522,"replication_time_us":15133}],["op",{"apply.queue_time_us":50,"apply.run_cpu_time_us":336,"apply.run_wall_time_us":336,"num_ops":1,"prepare.queue_time_us":429,"prepare.run_cpu_time_us":462,"prepare.run_wall_time_us":462,"replication_time_us":27132,"spinlock_wait_cycles":24320}],["op",{"apply.queue_time_us":4026,"apply.run_cpu_time_us":313,"apply.run_wall_time_us":776,"num_ops":1,"prepare.queue_time_us":2817,"prepare.run_cpu_time_us":462,"prepare.run_wall_time_us":461,"replication_time_us":9953}]]}
I20250112 02:52:44.975450 3110 ts_tablet_manager.cc:1918] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:44.975590 3110 log.cc:1198] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/3d662e0ce13f447595e5f92b94de3018
I20250112 02:52:44.976040 3110 ts_tablet_manager.cc:1939] T 3d662e0ce13f447595e5f92b94de3018 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:44.977558 2978 raft_consensus.cc:491] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.977686 2978 raft_consensus.cc:513] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.978351 2978 leader_election.cc:290] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:44.978986 2179 raft_consensus.cc:491] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:44.979110 2179 raft_consensus.cc:513] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:44.979707 2179 leader_election.cc:290] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:44.981465 1177 tablet_bootstrap.cc:492] T 9bbef42e11a843b18ed582ab4d04a92a P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:44.981514 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3d291ca3494f05ac23c7fe293e0bd9" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:44.981629 1177 ts_tablet_manager.cc:1397] T 9bbef42e11a843b18ed582ab4d04a92a P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.139s user 0.032s sys 0.009s
I20250112 02:52:44.981691 1049 raft_consensus.cc:2463] T 2d3d291ca3494f05ac23c7fe293e0bd9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:44.982177 714 leader_election.cc:304] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:44.982646 2451 raft_consensus.cc:2798] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:44.982729 2451 raft_consensus.cc:491] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:44.982846 2451 raft_consensus.cc:3054] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:44.985456 1050 raft_consensus.cc:2463] T fd81852962fb43449e35ef45dc1daaad P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:44.988512 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125b12a2f9164653be79d3e6d8313d36" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:44.989185 714 leader_election.cc:304] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:44.989782 2179 raft_consensus.cc:2798] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:44.989885 2179 raft_consensus.cc:695] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:44.990110 2179 consensus_queue.cc:237] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:44.991384 991 leader_election.cc:343] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:44.992159 451 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 205) took 3144 ms. Trace:
I20250112 02:52:44.992645 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3d291ca3494f05ac23c7fe293e0bd9" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:44.992846 917 raft_consensus.cc:2463] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:44.992255 451 rpcz_store.cc:276] 0112 02:52:41.847267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.925350 (+3078083us) service_pool.cc:225] Handling call
0112 02:52:44.927191 (+ 1841us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:44.927822 (+ 631us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:44.959314 (+ 31492us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:44.992005 (+ 32691us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:44.992098 (+ 93us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:44.928530 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:44.928627 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:44.928632 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:44.928635 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:44.928710 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:44.928721 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:44.928725 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:44.928791 (+ 66us) tablet.cc:698] Row locks acquired
0112 02:52:44.928794 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:44.928824 (+ 30us) write_op.cc:270] Start()
0112 02:52:44.928853 (+ 29us) write_op.cc:276] Timestamp: P: 1736650364928774 usec, L: 0
0112 02:52:44.928857 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:44.929019 (+ 162us) log.cc:844] Serialized 433 byte log entry
0112 02:52:44.949058 (+ 20039us) op_driver.cc:464] REPLICATION: finished
0112 02:52:44.949152 (+ 94us) write_op.cc:301] APPLY: starting
0112 02:52:44.949165 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:44.949203 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:44.949225 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:44.949551 (+ 326us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:44.949581 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:44.949587 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:44.949645 (+ 58us) log.cc:844] Serialized 29 byte log entry
0112 02:52:44.949696 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.949710 (+ 14us) write_op.cc:454] Released schema lock
0112 02:52:44.949715 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:44.949769 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.949772 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":53,"apply.run_cpu_time_us":408,"apply.run_wall_time_us":612,"num_ops":2,"prepare.queue_time_us":294,"prepare.run_cpu_time_us":581,"prepare.run_wall_time_us":582,"raft.queue_time_us":406,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":226,"replication_time_us":20176}]]}
I20250112 02:52:44.993475 449 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 206) took 3146 ms. Trace:
I20250112 02:52:44.995155 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 8429a3d8636d4e2ca4c1191f85b9266b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:44.995608 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5b3c712c0f448ea9b14ab93dd731ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:44.996285 1181 tablet_bootstrap.cc:492] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:44.996506 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125b12a2f9164653be79d3e6d8313d36" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:44.996647 919 raft_consensus.cc:2463] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:44.996392 1181 ts_tablet_manager.cc:1397] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.057s user 0.006s sys 0.010s
I20250112 02:52:44.996830 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet b1d9fae290ec4281839991f9d40b105c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:28 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:44.996976 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet b124c15736084e6d8f69b4fb7c41cf39 with delete_type TABLET_DATA_DELETED (replaced by tablet 97c7ce72d02049089c653684c53cb53c) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:44.997515 1181 raft_consensus.cc:357] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.997746 1181 raft_consensus.cc:383] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:44.997802 1181 raft_consensus.cc:738] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:44.998036 1181 consensus_queue.cc:260] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:44.998641 1181 ts_tablet_manager.cc:1428] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:44.998878 1181 tablet_bootstrap.cc:492] T 4a8ad3cda8124845824024cb69ff54ba P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:44.998997 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a7717cdcb8a4f47b11ba7af54b57e3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:43 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:44.999950 2787 raft_consensus.cc:491] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.000047 2787 raft_consensus.cc:513] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.000515 2787 leader_election.cc:290] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:45.000495 1181 tablet_bootstrap.cc:654] T 4a8ad3cda8124845824024cb69ff54ba P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:45.000878 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acafc4f579534a69a8f307ca4cf72e12" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:45.001428 855 leader_election.cc:343] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.001668 3133 raft_consensus.cc:491] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.001802 3133 raft_consensus.cc:513] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:45.003341 713 leader_election.cc:343] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 3b9280db29e54d41af1bed2fa5aa5535
I20250112 02:52:45.003568 3133 leader_election.cc:290] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:45.004766 450 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:58959 (request call id 244) took 2222 ms (client timeout 1000 ms). Trace:
W20250112 02:52:45.005566 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 8429a3d8636d4e2ca4c1191f85b9266b because the tablet was not found. No further retry: Not found: Tablet not found: 8429a3d8636d4e2ca4c1191f85b9266b
W20250112 02:52:45.007239 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet bd5b3c712c0f448ea9b14ab93dd731ab because the tablet was not found. No further retry: Not found: Tablet not found: bd5b3c712c0f448ea9b14ab93dd731ab
I20250112 02:52:45.007233 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acafc4f579534a69a8f307ca4cf72e12" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:45.007627 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b1d9fae290ec4281839991f9d40b105c because the tablet was not found. No further retry: Not found: Tablet not found: b1d9fae290ec4281839991f9d40b105c
W20250112 02:52:45.007885 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b124c15736084e6d8f69b4fb7c41cf39 because the tablet was not found. No further retry: Not found: Tablet not found: b124c15736084e6d8f69b4fb7c41cf39
I20250112 02:52:44.993585 449 rpcz_store.cc:276] 0112 02:52:41.847386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.926182 (+3078796us) service_pool.cc:225] Handling call
0112 02:52:44.935144 (+ 8962us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:44.935571 (+ 427us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:44.962716 (+ 27145us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:44.993345 (+ 30629us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:44.993433 (+ 88us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:44.936133 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:44.936208 (+ 75us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:44.936213 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:44.936216 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:44.936281 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:44.936290 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:44.936294 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:44.936352 (+ 58us) tablet.cc:698] Row locks acquired
0112 02:52:44.936354 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:44.936388 (+ 34us) write_op.cc:270] Start()
0112 02:52:44.936412 (+ 24us) write_op.cc:276] Timestamp: P: 1736650364936328 usec, L: 0
0112 02:52:44.936415 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:44.936534 (+ 119us) log.cc:844] Serialized 433 byte log entry
0112 02:52:44.956655 (+ 20121us) op_driver.cc:464] REPLICATION: finished
0112 02:52:44.956797 (+ 142us) write_op.cc:301] APPLY: starting
0112 02:52:44.956809 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:44.956843 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:44.960306 (+ 3463us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:44.960409 (+ 103us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:44.960441 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:44.960447 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:44.962551 (+ 2104us) log.cc:844] Serialized 29 byte log entry
0112 02:52:44.962608 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.962626 (+ 18us) write_op.cc:454] Released schema lock
0112 02:52:44.962633 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:44.962706 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.962711 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":100,"apply.run_cpu_time_us":476,"apply.run_wall_time_us":5896,"num_ops":2,"prepare.queue_time_us":36,"prepare.run_cpu_time_us":455,"prepare.run_wall_time_us":455,"replication_time_us":20218}]]}
W20250112 02:52:45.008970 714 leader_election.cc:343] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 3b9280db29e54d41af1bed2fa5aa5535
I20250112 02:52:45.009152 714 leader_election.cc:304] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
W20250112 02:52:45.008109 450 rpcz_store.cc:269] 0112 02:52:42.781865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:43.248786 (+466921us) service_pool.cc:225] Handling call
0112 02:52:45.004722 (+1755936us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:43.909693 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:43.909780 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:43.909785 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:43.909787 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:43.909887 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:43.909898 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:43.909901 (+ 3us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:43.910012 (+ 111us) tablet.cc:698] Row locks acquired
0112 02:52:43.910015 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:43.910044 (+ 29us) write_op.cc:270] Start()
0112 02:52:43.910073 (+ 29us) write_op.cc:276] Timestamp: P: 1736650363909994 usec, L: 0
0112 02:52:43.910076 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:43.910280 (+ 204us) log.cc:844] Serialized 901 byte log entry
0112 02:52:43.913951 (+ 3671us) op_driver.cc:464] REPLICATION: finished
0112 02:52:43.914037 (+ 86us) write_op.cc:301] APPLY: starting
0112 02:52:43.914049 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:43.914086 (+ 37us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:43.914106 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:43.914165 (+ 59us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:43.914197 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:43.914204 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:43.914263 (+ 59us) log.cc:844] Serialized 29 byte log entry
0112 02:52:43.914309 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:43.914326 (+ 17us) write_op.cc:454] Released schema lock
0112 02:52:43.914332 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:43.914388 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:43.914393 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":25,"reported_tablets":174,"child_traces":[["op",{"apply.queue_time_us":41,"apply.run_cpu_time_us":345,"apply.run_wall_time_us":345,"num_ops":2,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":675,"prepare.run_wall_time_us":675,"raft.queue_time_us":736,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":210,"replication_time_us":3850}]]}
I20250112 02:52:45.010001 2179 raft_consensus.cc:2743] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:45.011065 989 leader_election.cc:304] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:45.011286 2745 raft_consensus.cc:2798] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:45.011343 2745 raft_consensus.cc:491] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:45.011391 2745 raft_consensus.cc:3054] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.012370 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_b32fcf8b32d44d2093696313ffcb2083"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.012555 854 leader_election.cc:343] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.012657 854 leader_election.cc:304] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:45.012877 2787 raft_consensus.cc:2743] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:45.012923 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b32fcf8b32d44d2093696313ffcb2083 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.013444 1181 tablet_bootstrap.cc:492] T 4a8ad3cda8124845824024cb69ff54ba P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:45.013564 1181 ts_tablet_manager.cc:1397] T 4a8ad3cda8124845824024cb69ff54ba P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.004s
I20250112 02:52:45.013808 1177 raft_consensus.cc:357] T 9bbef42e11a843b18ed582ab4d04a92a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.014159 1177 raft_consensus.cc:383] T 9bbef42e11a843b18ed582ab4d04a92a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:45.014145 1181 raft_consensus.cc:357] T 4a8ad3cda8124845824024cb69ff54ba P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.014302 1181 raft_consensus.cc:383] T 4a8ad3cda8124845824024cb69ff54ba P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:45.014355 1181 raft_consensus.cc:738] T 4a8ad3cda8124845824024cb69ff54ba P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:45.014326 1177 raft_consensus.cc:738] T 9bbef42e11a843b18ed582ab4d04a92a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:45.014575 1181 consensus_queue.cc:260] T 4a8ad3cda8124845824024cb69ff54ba P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.015131 1181 ts_tablet_manager.cc:1428] T 4a8ad3cda8124845824024cb69ff54ba P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250112 02:52:45.015031 1177 consensus_queue.cc:260] T 9bbef42e11a843b18ed582ab4d04a92a P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.015491 1181 tablet_bootstrap.cc:492] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:45.015791 1177 ts_tablet_manager.cc:1428] T 9bbef42e11a843b18ed582ab4d04a92a P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.034s user 0.001s sys 0.001s
I20250112 02:52:45.016434 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_0be1f83063bb4449a2129e293757be37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.016891 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0be1f83063bb4449a2129e293757be37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.018682 1181 tablet_bootstrap.cc:654] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:45.025475 451 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 208) took 2895 ms. Trace:
I20250112 02:52:45.025528 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 68e480204ee94a44a1fef701ef5541f7 (DEFAULT_TABLE table=default.Table_54e45ab6e6d14005bfe1cc06501f6d9a [id=3dcd0dbb9dcc445eaec3448a6cf96163]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.025882 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68e480204ee94a44a1fef701ef5541f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.025563 451 rpcz_store.cc:276] 0112 02:52:42.129570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.993126 (+2863556us) service_pool.cc:225] Handling call
0112 02:52:45.012262 (+ 19136us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd72d03 00007f3e13e75236 00007f3e13e74f65 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:45.013002 (+ 740us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:45.013432 (+ 430us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:45.025216 (+ 11784us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:45.025346 (+ 130us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:45.025420 (+ 74us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:45.014960 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:45.015042 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:45.015050 (+ 8us) write_op.cc:435] Acquired schema lock
0112 02:52:45.015054 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:45.015161 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:45.015168 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:45.015173 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:45.015239 (+ 66us) tablet.cc:698] Row locks acquired
0112 02:52:45.015243 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:45.015270 (+ 27us) write_op.cc:270] Start()
0112 02:52:45.015294 (+ 24us) write_op.cc:276] Timestamp: P: 1736650365015203 usec, L: 0
0112 02:52:45.015296 (+ 2us) op_driver.cc:348] REPLICATION: starting
0112 02:52:45.015431 (+ 135us) log.cc:844] Serialized 433 byte log entry
0112 02:52:45.021246 (+ 5815us) op_driver.cc:464] REPLICATION: finished
0112 02:52:45.022175 (+ 929us) write_op.cc:301] APPLY: starting
0112 02:52:45.022184 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:45.022209 (+ 25us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:45.022224 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:45.022326 (+ 102us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:45.022351 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:45.022357 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:45.022424 (+ 67us) log.cc:844] Serialized 29 byte log entry
0112 02:52:45.022461 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.022475 (+ 14us) write_op.cc:454] Released schema lock
0112 02:52:45.022480 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":43268480,"child_traces":[["op",{"apply.queue_time_us":901,"apply.run_cpu_time_us":357,"apply.run_wall_time_us":357,"num_ops":2,"prepare.queue_time_us":1110,"raft.queue_time_us":2604,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":2,"replication_time_us":5940,"threads_started":1}]]}
I20250112 02:52:45.026145 451 master_service.cc:946] ReplaceTablet: received request to replace tablet ea33053c62dc4707b759d5c1756ab4bd from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:45.028473 890 tablet_service.cc:1467] Processing CreateTablet for tablet 68e480204ee94a44a1fef701ef5541f7 (DEFAULT_TABLE table=default.Table_54e45ab6e6d14005bfe1cc06501f6d9a [id=3dcd0dbb9dcc445eaec3448a6cf96163]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.028867 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68e480204ee94a44a1fef701ef5541f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.030683 1034 tablet_service.cc:1467] Processing CreateTablet for tablet fc67986a45ed414296e68efac8e48fbf (DEFAULT_TABLE table=default.Table_3406d849cc9a4ffab1b431da38b95bcf [id=edd75218d8224da0b59ccffc63d2b49f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.031109 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc67986a45ed414296e68efac8e48fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.031744 897 tablet_service.cc:1467] Processing CreateTablet for tablet fc67986a45ed414296e68efac8e48fbf (DEFAULT_TABLE table=default.Table_3406d849cc9a4ffab1b431da38b95bcf [id=edd75218d8224da0b59ccffc63d2b49f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.032136 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc67986a45ed414296e68efac8e48fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.035727 898 tablet_service.cc:1467] Processing CreateTablet for tablet aa02a8131c70402cb319355f9ad9bd27 (DEFAULT_TABLE table=default.Table_31ea460b9f1d4424932127e3eb550a27 [id=a98e249c051b4e699da205aad4177a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.036053 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa02a8131c70402cb319355f9ad9bd27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.036597 1017 tablet_service.cc:1467] Processing CreateTablet for tablet aa02a8131c70402cb319355f9ad9bd27 (DEFAULT_TABLE table=default.Table_31ea460b9f1d4424932127e3eb550a27 [id=a98e249c051b4e699da205aad4177a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.036880 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa02a8131c70402cb319355f9ad9bd27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.041065 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:45.041909 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_5d473cabff2e46e0b531fbce28d9d558 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 1 got 0
I20250112 02:52:45.042074 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:45.042374 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_a9907ed8bfe14625977ec98b67e9d599 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 2 got 0
I20250112 02:52:45.042549 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_b9dff6cb7bbd4d0a88de458399b0d2c0 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 2 got 0
I20250112 02:52:45.043022 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 37d38f5e525440f8a0f7822620933435 (table default.Table_5aeb2d7bb68c46a589a48526dcc40aaf [id=9d267774bf59428e854651956fbd79be]). Expected version 1 got 0
I20250112 02:52:45.043540 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_87c7c83e6cef4175a123fa9924acd6d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.044073 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87c7c83e6cef4175a123fa9924acd6d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.046082 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 3d662e0ce13f447595e5f92b94de3018 (table default.Table_498525c117bf4424a217c15eda6ba3ce [id=87b950688174482d8509a759d3c6e28f]) successfully deleted
I20250112 02:52:45.046123 2787 raft_consensus.cc:491] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.046216 2787 raft_consensus.cc:513] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.045951 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_18c158c848f1430f9852655c57f10933"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.046635 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18c158c848f1430f9852655c57f10933 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.046694 3110 tablet_replica.cc:331] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:45.046733 2787 leader_election.cc:290] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:45.046891 3110 raft_consensus.cc:2238] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:45.047017 3110 raft_consensus.cc:2267] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:45.047084 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet b124c15736084e6d8f69b4fb7c41cf39 because the tablet was not found. No further retry: Not found: Tablet not found: b124c15736084e6d8f69b4fb7c41cf39
W20250112 02:52:45.047305 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet ff1b3ee229c445b19405cf4a19cf8003 because the tablet was not found. No further retry: Not found: Tablet not found: ff1b3ee229c445b19405cf4a19cf8003
I20250112 02:52:45.047452 3110 ts_tablet_manager.cc:1905] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:45.047520 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 23e7a2e88c374e949ced92c557225cbf because the tablet was not found. No further retry: Not found: Tablet not found: 23e7a2e88c374e949ced92c557225cbf
W20250112 02:52:45.047683 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 3b9280db29e54d41af1bed2fa5aa5535 because the tablet was not found. No further retry: Not found: Tablet not found: 3b9280db29e54d41af1bed2fa5aa5535
W20250112 02:52:45.047825 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 83c04f1a2cb04950a9f6170e252c19b4 because the tablet was not found. No further retry: Not found: Tablet not found: 83c04f1a2cb04950a9f6170e252c19b4
W20250112 02:52:45.047971 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 795ac7eaf81d48bb926865d9ebb388b6 because the tablet was not found. No further retry: Not found: Tablet not found: 795ac7eaf81d48bb926865d9ebb388b6
W20250112 02:52:45.048120 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 781a9943f705477cb6eae16df4468d2f because the tablet was not found. No further retry: Not found: Tablet not found: 781a9943f705477cb6eae16df4468d2f
I20250112 02:52:45.051277 2985 raft_consensus.cc:491] T 5a3af0ea4e954053ae89814f3e74e6b7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.051396 2985 raft_consensus.cc:513] T 5a3af0ea4e954053ae89814f3e74e6b7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.052026 2985 leader_election.cc:290] T 5a3af0ea4e954053ae89814f3e74e6b7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:45.052273 1181 tablet_bootstrap.cc:492] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:45.052376 1181 ts_tablet_manager.cc:1397] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.000s
I20250112 02:52:45.053177 1181 raft_consensus.cc:357] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.053336 1181 raft_consensus.cc:383] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:45.053388 1181 raft_consensus.cc:738] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:45.053586 1181 consensus_queue.cc:260] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.054579 1181 ts_tablet_manager.cc:1428] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:45.054984 1181 tablet_bootstrap.cc:492] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:45.055058 2985 raft_consensus.cc:491] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.055971 2985 raft_consensus.cc:513] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.056311 1181 tablet_bootstrap.cc:654] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:45.056524 2985 leader_election.cc:290] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:45.057796 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a3af0ea4e954053ae89814f3e74e6b7" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:45.058436 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5524d75599f04becbe6a94426adb7026" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:45.058938 989 leader_election.cc:343] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 058a9ca089cd4967887b0a1127c7d03f
W20250112 02:52:45.059146 854 leader_election.cc:343] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.063633 1173 tablet_bootstrap.cc:492] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:45.063812 1173 ts_tablet_manager.cc:1397] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.143s user 0.001s sys 0.003s
I20250112 02:52:45.064015 447 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 207) took 3199 ms. Trace:
I20250112 02:52:45.030463 743 tablet_service.cc:1467] Processing CreateTablet for tablet aa02a8131c70402cb319355f9ad9bd27 (DEFAULT_TABLE table=default.Table_31ea460b9f1d4424932127e3eb550a27 [id=a98e249c051b4e699da205aad4177a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.039813 756 tablet_service.cc:1467] Processing CreateTablet for tablet 68e480204ee94a44a1fef701ef5541f7 (DEFAULT_TABLE table=default.Table_54e45ab6e6d14005bfe1cc06501f6d9a [id=3dcd0dbb9dcc445eaec3448a6cf96163]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.064136 447 rpcz_store.cc:276] 0112 02:52:41.864833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.974017 (+3109184us) service_pool.cc:225] Handling call
0112 02:52:45.016334 (+ 42317us) spinlock_profiling.cc:243] Waited 42.1 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd72d03 00007f3e13e75236 00007f3e13e74f65 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:45.016970 (+ 636us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:45.017353 (+ 383us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:45.049499 (+ 32146us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:45.063851 (+ 14352us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:45.063956 (+ 105us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:45.043537 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:45.043651 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:45.043658 (+ 7us) write_op.cc:435] Acquired schema lock
0112 02:52:45.043662 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:45.043768 (+ 106us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:45.043783 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:45.043790 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:45.043875 (+ 85us) tablet.cc:698] Row locks acquired
0112 02:52:45.043879 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:45.043910 (+ 31us) write_op.cc:270] Start()
0112 02:52:45.043940 (+ 30us) write_op.cc:276] Timestamp: P: 1736650365043835 usec, L: 0
0112 02:52:45.043962 (+ 22us) op_driver.cc:348] REPLICATION: starting
0112 02:52:45.044139 (+ 177us) log.cc:844] Serialized 433 byte log entry
0112 02:52:45.047118 (+ 2979us) op_driver.cc:464] REPLICATION: finished
0112 02:52:45.048847 (+ 1729us) write_op.cc:301] APPLY: starting
0112 02:52:45.048859 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:45.048891 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:45.048949 (+ 58us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:45.049082 (+ 133us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:45.049110 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:45.049117 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:45.049205 (+ 88us) log.cc:844] Serialized 29 byte log entry
0112 02:52:45.049412 (+ 207us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.049426 (+ 14us) write_op.cc:454] Released schema lock
0112 02:52:45.049432 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:45.049928 (+ 496us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.049933 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":96735488,"child_traces":[["op",{"apply.queue_time_us":1667,"apply.run_cpu_time_us":524,"apply.run_wall_time_us":778,"num_ops":2,"prepare.queue_time_us":25753,"prepare.run_cpu_time_us":713,"prepare.run_wall_time_us":6396,"raft.queue_time_us":32,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":204,"replication_time_us":3132}]]}
I20250112 02:52:45.040161 747 tablet_service.cc:1467] Processing CreateTablet for tablet fc67986a45ed414296e68efac8e48fbf (DEFAULT_TABLE table=default.Table_3406d849cc9a4ffab1b431da38b95bcf [id=edd75218d8224da0b59ccffc63d2b49f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.064754 447 master_service.cc:946] ReplaceTablet: received request to replace tablet a4328137407843e1879493ef0c56ffc8 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:45.040442 2179 raft_consensus.cc:491] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.049046 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5524d75599f04becbe6a94426adb7026" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:45.064978 449 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 209) took 2929 ms. Trace:
I20250112 02:52:45.064954 1173 raft_consensus.cc:357] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:45.065204 855 leader_election.cc:343] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.065218 1173 raft_consensus.cc:383] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:45.065294 855 leader_election.cc:304] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:45.065340 1173 raft_consensus.cc:738] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:45.065104 449 rpcz_store.cc:276] 0112 02:52:42.135298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.008739 (+2873441us) service_pool.cc:225] Handling call
0112 02:52:45.043427 (+ 34688us) spinlock_profiling.cc:243] Waited 34.4 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd72d03 00007f3e13e75236 00007f3e13e74f65 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:45.044154 (+ 727us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:45.044572 (+ 418us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:45.056578 (+ 12006us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:45.064861 (+ 8283us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:45.064935 (+ 74us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:45.050035 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:45.050152 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:45.050157 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:45.050159 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:45.050255 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:45.050264 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:45.050286 (+ 22us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:45.050441 (+ 155us) tablet.cc:698] Row locks acquired
0112 02:52:45.050445 (+ 4us) write_op.cc:260] PREPARE: finished
0112 02:52:45.050495 (+ 50us) write_op.cc:270] Start()
0112 02:52:45.050522 (+ 27us) write_op.cc:276] Timestamp: P: 1736650365050419 usec, L: 0
0112 02:52:45.050525 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:45.050734 (+ 209us) log.cc:844] Serialized 433 byte log entry
0112 02:52:45.053625 (+ 2891us) op_driver.cc:464] REPLICATION: finished
0112 02:52:45.055649 (+ 2024us) write_op.cc:301] APPLY: starting
0112 02:52:45.055683 (+ 34us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:45.055828 (+ 145us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:45.055856 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:45.056042 (+ 186us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:45.056072 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:45.056081 (+ 9us) write_op.cc:312] APPLY: finished
0112 02:52:45.056178 (+ 97us) log.cc:844] Serialized 29 byte log entry
0112 02:52:45.056427 (+ 249us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.056466 (+ 39us) write_op.cc:454] Released schema lock
0112 02:52:45.056472 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:45.056887 (+ 415us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.056892 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":79119104,"child_traces":[["op",{"apply.queue_time_us":1975,"apply.run_cpu_time_us":591,"apply.run_wall_time_us":919,"num_ops":2,"prepare.queue_time_us":5079,"prepare.run_cpu_time_us":686,"prepare.run_wall_time_us":6864,"raft.queue_time_us":44,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":4237,"replication_time_us":3077}]]}
I20250112 02:52:45.065613 2787 raft_consensus.cc:2743] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:45.065821 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_5981a14692a2470290628d9e5934c4e5" } new_table_name: "default.Table_9ecfcdf0c66d4e3184fae185f8fe6c9e" modify_external_catalogs: true
I20250112 02:52:45.066097 1173 consensus_queue.cc:260] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.066694 1173 ts_tablet_manager.cc:1428] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 02:52:45.066953 1173 tablet_bootstrap.cc:492] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:45.067389 1181 tablet_bootstrap.cc:492] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:45.067572 1181 ts_tablet_manager.cc:1397] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250112 02:52:45.068121 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]). Expected version 2 got 0
I20250112 02:52:45.068457 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ca2165362ee541adac724b63f39b1489 (table default.Table_41b7056b6e9240928d0af52aec2e9338 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 2 got 0
I20250112 02:52:45.068526 1181 raft_consensus.cc:357] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.068789 1181 raft_consensus.cc:383] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:45.068895 1181 raft_consensus.cc:738] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:45.069191 1181 consensus_queue.cc:260] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.071995 450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 210) took 2933 ms. Trace:
I20250112 02:52:45.072077 450 rpcz_store.cc:276] 0112 02:52:42.138095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.011253 (+2873158us) service_pool.cc:225] Handling call
0112 02:52:45.045874 (+ 34621us) spinlock_profiling.cc:243] Waited 33.7 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd72d03 00007f3e13e75236 00007f3e13e74f65 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:45.063761 (+ 17887us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:45.064215 (+ 454us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:45.071778 (+ 7563us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:45.071883 (+ 105us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:45.071948 (+ 65us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:45.066839 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:45.066910 (+ 71us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:45.066916 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:45.066919 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:45.066990 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:45.066999 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:45.067002 (+ 3us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:45.067059 (+ 57us) tablet.cc:698] Row locks acquired
0112 02:52:45.067061 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:45.067086 (+ 25us) write_op.cc:270] Start()
0112 02:52:45.067112 (+ 26us) write_op.cc:276] Timestamp: P: 1736650365067009 usec, L: 0
0112 02:52:45.067116 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:45.067241 (+ 125us) log.cc:844] Serialized 433 byte log entry
0112 02:52:45.070838 (+ 3597us) op_driver.cc:464] REPLICATION: finished
0112 02:52:45.070926 (+ 88us) write_op.cc:301] APPLY: starting
0112 02:52:45.070936 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:45.070968 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:45.070994 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:45.071071 (+ 77us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:45.071097 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:45.071103 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:45.071162 (+ 59us) log.cc:844] Serialized 29 byte log entry
0112 02:52:45.071204 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.071217 (+ 13us) write_op.cc:454] Released schema lock
0112 02:52:45.071221 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:45.071271 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.071275 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":77573504,"child_traces":[["op",{"apply.queue_time_us":53,"apply.run_cpu_time_us":341,"apply.run_wall_time_us":340,"num_ops":2,"prepare.queue_time_us":2278,"prepare.run_cpu_time_us":485,"prepare.run_wall_time_us":484,"raft.queue_time_us":315,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":293,"replication_time_us":3700}]]}
I20250112 02:52:45.072633 1181 ts_tablet_manager.cc:1428] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 02:52:45.072738 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_70d1488de98d4380b8024db852f58991" } new_table_name: "default.Table_dc292752faf041ed8ca40958c86765b0" modify_external_catalogs: true
I20250112 02:52:45.073019 1181 tablet_bootstrap.cc:492] T 7eb24b3d8a9446e8abd9729462225002 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:45.075925 1181 tablet_bootstrap.cc:654] T 7eb24b3d8a9446e8abd9729462225002 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:45.078285 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ba2bcb3825c0471a9bff813594d354f9 [id=097d902a92934882bc7c016da518642a]). Expected version 1 got 0
I20250112 02:52:45.078572 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:45.078753 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 3dab0466476c4a9aa265271c1abd91a4 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:45.079228 448 catalog_manager.cc:5526] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:45.079666 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:45.079885 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 56ddc025a8314f6f84d229191eb56692 (table default.Table_d22db1cc1d354af495d9888de1bfc22a [id=2ddd5dd296b94bc8b96a97b66e775836]). Expected version 1 got 0
I20250112 02:52:45.081441 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:45.081727 448 catalog_manager.cc:5526] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:45.082477 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5524d75599f04becbe6a94426adb7026 (table default.Table_e8cef38ebaa243ed83bfb41bcea1933f [id=b7739f28885a453d9ef3966511183c17]). Expected version 1 got 0
I20250112 02:52:45.082917 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:45.083884 1181 tablet_bootstrap.cc:492] T 7eb24b3d8a9446e8abd9729462225002 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:45.084004 1181 ts_tablet_manager.cc:1397] T 7eb24b3d8a9446e8abd9729462225002 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20250112 02:52:45.084546 1181 raft_consensus.cc:357] T 7eb24b3d8a9446e8abd9729462225002 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.084698 1181 raft_consensus.cc:383] T 7eb24b3d8a9446e8abd9729462225002 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:45.084746 1181 raft_consensus.cc:738] T 7eb24b3d8a9446e8abd9729462225002 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:45.087149 1181 consensus_queue.cc:260] T 7eb24b3d8a9446e8abd9729462225002 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.087641 1181 ts_tablet_manager.cc:1428] T 7eb24b3d8a9446e8abd9729462225002 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:45.087845 1181 tablet_bootstrap.cc:492] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:45.089013 1181 tablet_bootstrap.cc:654] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:45.089614 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_c3651f53d9004655b02efca2428b3ead [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 2 got 0
I20250112 02:52:45.090000 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:45.090346 2179 raft_consensus.cc:513] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.090539 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet d95f98ae7bdf42518bbff55e257fea88 (table default.Table_2fcdbb05970f4a098f7ba3e287fa0348 [id=70c61753e9944e1fbd65e9ec41a48ee2]). Expected version 1 got 0
I20250112 02:52:45.090729 3133 consensus_queue.cc:1035] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20250112 02:52:45.090788 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:45.091284 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:45.091318 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058a9ca089cd4967887b0a1127c7d03f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:45.090313 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa02a8131c70402cb319355f9ad9bd27. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:45.092168 991 leader_election.cc:343] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.092227 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 17c54c5dc86446cf85974777c8145ef6 (table default.Table_8ab566cf9c8f45eba4dc9bb85264c823 [id=a622f7a973344a388b3275adbcce6234]). Expected version 1 got 0
I20250112 02:52:45.092301 991 leader_election.cc:304] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:45.092736 2985 raft_consensus.cc:2743] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:45.093210 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cd52261c4a414e7d93a528e2eddfa9db consensus state has no leader
I20250112 02:52:45.093389 450 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 214) took 1629 ms. Trace:
I20250112 02:52:45.093469 450 rpcz_store.cc:276] 0112 02:52:43.464307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.072599 (+1608292us) service_pool.cc:225] Handling call
0112 02:52:45.072899 (+ 300us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:45.073089 (+ 190us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:45.073233 (+ 144us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:45.073319 (+ 86us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:45.082628 (+ 9309us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:45.093329 (+ 10701us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:45.076065 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:45.076160 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:45.076164 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:45.076166 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:45.076228 (+ 62us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:45.076235 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:45.076239 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:45.076279 (+ 40us) tablet.cc:698] Row locks acquired
0112 02:52:45.076281 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:45.076307 (+ 26us) write_op.cc:270] Start()
0112 02:52:45.076328 (+ 21us) write_op.cc:276] Timestamp: P: 1736650365076226 usec, L: 0
0112 02:52:45.076331 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:45.076479 (+ 148us) log.cc:844] Serialized 405 byte log entry
0112 02:52:45.080716 (+ 4237us) op_driver.cc:464] REPLICATION: finished
0112 02:52:45.080802 (+ 86us) write_op.cc:301] APPLY: starting
0112 02:52:45.080812 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:45.080833 (+ 21us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:45.080873 (+ 40us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:45.080910 (+ 37us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:45.080940 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:45.080950 (+ 10us) write_op.cc:312] APPLY: finished
0112 02:52:45.080999 (+ 49us) log.cc:844] Serialized 23 byte log entry
0112 02:52:45.081039 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.081051 (+ 12us) write_op.cc:454] Released schema lock
0112 02:52:45.081055 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:45.081121 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.081124 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":46,"apply.run_cpu_time_us":294,"apply.run_wall_time_us":294,"num_ops":1,"prepare.queue_time_us":2381,"prepare.run_cpu_time_us":468,"prepare.run_wall_time_us":468,"replication_time_us":4365}]]}
I20250112 02:52:45.094209 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_ba2bcb3825c0471a9bff813594d354f9" } new_table_name: "default.Table_ce6e0afd36bf48c4937196300248bc26" modify_external_catalogs: true
I20250112 02:52:45.094261 3109 ts_tablet_manager.cc:1918] T a3ffdc237f8640f0b378809f93fc6292 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:45.095546 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68e480204ee94a44a1fef701ef5541f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.090313 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a3af0ea4e954053ae89814f3e74e6b7" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:45.098549 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b455a37448a044679bd2c0646de3d804 consensus state has no leader
I20250112 02:52:45.098671 449 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 213) took 1977 ms. Trace:
I20250112 02:52:45.098757 449 rpcz_store.cc:276] 0112 02:52:43.121523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.065647 (+1944124us) service_pool.cc:225] Handling call
0112 02:52:45.066002 (+ 355us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:45.066208 (+ 206us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:45.066394 (+ 186us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:45.066468 (+ 74us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:45.077759 (+ 11291us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:45.098648 (+ 20889us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:45.067372 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:45.067476 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:45.067482 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:45.067485 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:45.067556 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:45.067563 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:45.067568 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:45.067621 (+ 53us) tablet.cc:698] Row locks acquired
0112 02:52:45.067623 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:45.067658 (+ 35us) write_op.cc:270] Start()
0112 02:52:45.067680 (+ 22us) write_op.cc:276] Timestamp: P: 1736650365067582 usec, L: 0
0112 02:52:45.067683 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:45.067813 (+ 130us) log.cc:844] Serialized 405 byte log entry
0112 02:52:45.077377 (+ 9564us) op_driver.cc:464] REPLICATION: finished
0112 02:52:45.077457 (+ 80us) write_op.cc:301] APPLY: starting
0112 02:52:45.077467 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:45.077490 (+ 23us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:45.077508 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:45.077540 (+ 32us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:45.077563 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:45.077569 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:45.077618 (+ 49us) log.cc:844] Serialized 23 byte log entry
0112 02:52:45.077666 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.077678 (+ 12us) write_op.cc:454] Released schema lock
0112 02:52:45.077682 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:45.077736 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.077739 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":46,"apply.run_cpu_time_us":272,"apply.run_wall_time_us":271,"num_ops":1,"prepare.queue_time_us":551,"prepare.run_cpu_time_us":520,"prepare.run_wall_time_us":519,"replication_time_us":9665,"spinlock_wait_cycles":4352}]]}
I20250112 02:52:45.099339 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_5d473cabff2e46e0b531fbce28d9d558" } new_table_name: "default.Table_ee16ba338f554f4dbd75b332adf897f3" modify_external_catalogs: true
I20250112 02:52:45.092938 2451 raft_consensus.cc:513] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.100549 1177 tablet_bootstrap.cc:492] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:45.100594 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062] alter complete (version 1)
I20250112 02:52:45.107018 991 leader_election.cc:304] T 5a3af0ea4e954053ae89814f3e74e6b7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
W20250112 02:52:45.110042 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca6f080e5fd84084b601b646d4e6c9e2 consensus state has no leader
I20250112 02:52:45.110169 449 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 216) took 1645 ms. Trace:
I20250112 02:52:45.110251 449 rpcz_store.cc:276] 0112 02:52:43.464840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.099214 (+1634374us) service_pool.cc:225] Handling call
0112 02:52:45.099491 (+ 277us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:45.099666 (+ 175us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:45.099788 (+ 122us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:45.099834 (+ 46us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:45.108446 (+ 8612us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:45.110143 (+ 1697us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:45.100349 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:45.100434 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:45.100441 (+ 7us) write_op.cc:435] Acquired schema lock
0112 02:52:45.100444 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:45.100528 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:45.100539 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:45.100545 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:45.100625 (+ 80us) tablet.cc:698] Row locks acquired
0112 02:52:45.100627 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:45.100658 (+ 31us) write_op.cc:270] Start()
0112 02:52:45.100682 (+ 24us) write_op.cc:276] Timestamp: P: 1736650365100571 usec, L: 0
0112 02:52:45.100685 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:45.100795 (+ 110us) log.cc:844] Serialized 405 byte log entry
0112 02:52:45.107627 (+ 6832us) op_driver.cc:464] REPLICATION: finished
0112 02:52:45.107782 (+ 155us) write_op.cc:301] APPLY: starting
0112 02:52:45.107793 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:45.107817 (+ 24us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:45.107835 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:45.107869 (+ 34us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:45.107892 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:45.107898 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:45.107952 (+ 54us) log.cc:844] Serialized 23 byte log entry
0112 02:52:45.107999 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.108010 (+ 11us) write_op.cc:454] Released schema lock
0112 02:52:45.108015 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:45.108069 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.108073 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":113,"apply.run_cpu_time_us":284,"apply.run_wall_time_us":284,"num_ops":1,"prepare.queue_time_us":132,"prepare.run_cpu_time_us":500,"prepare.run_wall_time_us":500,"replication_time_us":6917}]]}
I20250112 02:52:45.110852 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_489753b0c1154a4ba82ad30de84de181" } modify_external_catalogs: true
I20250112 02:52:45.111006 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_489753b0c1154a4ba82ad30de84de181" } modify_external_catalogs: true
I20250112 02:52:45.112046 450 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40610 (request call id 215) took 1647 ms. Trace:
I20250112 02:52:45.112169 450 rpcz_store.cc:276] 0112 02:52:43.464422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.094041 (+1629619us) service_pool.cc:225] Handling call
0112 02:52:45.094390 (+ 349us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:45.094623 (+ 233us) catalog_manager.cc:3627] Apply alter schema
0112 02:52:45.094790 (+ 167us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:52:45.094852 (+ 62us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:52:45.110950 (+ 16098us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:52:45.112005 (+ 1055us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:45.095557 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:45.095635 (+ 78us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:45.095640 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:45.095643 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:45.095716 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:45.095726 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:45.095730 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:45.095778 (+ 48us) tablet.cc:698] Row locks acquired
0112 02:52:45.095780 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:45.095807 (+ 27us) write_op.cc:270] Start()
0112 02:52:45.095830 (+ 23us) write_op.cc:276] Timestamp: P: 1736650365095724 usec, L: 0
0112 02:52:45.095833 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:45.095960 (+ 127us) log.cc:844] Serialized 405 byte log entry
0112 02:52:45.099079 (+ 3119us) op_driver.cc:464] REPLICATION: finished
0112 02:52:45.099164 (+ 85us) write_op.cc:301] APPLY: starting
0112 02:52:45.099174 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:45.099201 (+ 27us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:45.099227 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:45.099262 (+ 35us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:45.099285 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:45.099291 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:45.099343 (+ 52us) log.cc:844] Serialized 23 byte log entry
0112 02:52:45.099402 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.099416 (+ 14us) write_op.cc:454] Released schema lock
0112 02:52:45.099424 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:45.099500 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.099504 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":53,"apply.run_cpu_time_us":327,"apply.run_wall_time_us":326,"num_ops":1,"prepare.queue_time_us":250,"prepare.run_cpu_time_us":488,"prepare.run_wall_time_us":489,"raft.queue_time_us":529,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":191,"replication_time_us":3224}]]}
I20250112 02:52:45.112881 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_2989b0fb83514c45ac9836e95e444f31" } modify_external_catalogs: true
I20250112 02:52:45.113056 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_2989b0fb83514c45ac9836e95e444f31" } modify_external_catalogs: true
W20250112 02:52:45.114431 991 consensus_peers.cc:487] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:52:45.115629 454 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:38361 (request call id 218) took 1357 ms (client timeout 1000 ms). Trace:
I20250112 02:52:45.118036 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 2361bc6bd8164c2daa37be6b0845f4bd with delete_type TABLET_DATA_DELETED (replaced by tablet 154ecbf4175a42a7ad765780c0f902a5) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:45.118281 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1954e7859f422f9690fe4a9384a8bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:45.118474 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b351c563d749df806ab5d0331a4ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:45.118669 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:45.118841 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f4bd2df0ce45dabc55373f1154d3ac with delete_type TABLET_DATA_DELETED (replaced by tablet f559dedfb7854cf890bef362c50b4213) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:45.119369 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_6113c9f3ad094f0381176b06d637966c [id=08a760dcd20642fb869437d4ccc3f333] alter complete (version 1)
I20250112 02:52:45.120011 3111 tablet_replica.cc:331] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:45.120208 3111 raft_consensus.cc:2238] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:45.120380 3111 raft_consensus.cc:2267] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:45.120913 3111 ts_tablet_manager.cc:1905] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:45.118985 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 54215c55966a46e1ab96e352c4e3764e with delete_type TABLET_DATA_DELETED (replaced by tablet a470f84d50df4cf0b4a1e5d248b4c32a) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:45.123999 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 2361bc6bd8164c2daa37be6b0845f4bd because the tablet was not found. No further retry: Not found: Tablet not found: 2361bc6bd8164c2daa37be6b0845f4bd
I20250112 02:52:45.124943 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_556e3fb3ccfb4128b7224a024ad819ac [id=3a629cb4a4104e30b0beb8f729ba0a22] alter complete (version 1)
W20250112 02:52:45.115725 454 rpcz_store.cc:269] 0112 02:52:43.758381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:43.926316 (+167935us) service_pool.cc:225] Handling call
0112 02:52:44.089473 (+163157us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.089478 (+ 5us) write_op.cc:454] Released schema lock
0112 02:52:44.279450 (+189972us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.279458 (+ 8us) write_op.cc:454] Released schema lock
0112 02:52:45.115572 (+836114us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:44.081788 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:44.081887 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:44.081893 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:44.081896 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:44.082071 (+ 175us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:44.082083 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:44.082087 (+ 4us) tablet.cc:682] Acquiring locks for 5 operations
0112 02:52:44.082253 (+ 166us) tablet.cc:698] Row locks acquired
0112 02:52:44.082256 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:44.082289 (+ 33us) write_op.cc:270] Start()
0112 02:52:44.082320 (+ 31us) write_op.cc:276] Timestamp: P: 1736650364082203 usec, L: 0
0112 02:52:44.082323 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:44.082487 (+ 164us) log.cc:844] Serialized 2024 byte log entry
0112 02:52:44.086415 (+ 3928us) op_driver.cc:464] REPLICATION: finished
0112 02:52:44.086558 (+ 143us) write_op.cc:301] APPLY: starting
0112 02:52:44.086571 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:44.086650 (+ 79us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:44.086678 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:44.086777 (+ 99us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:44.086809 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0112 02:52:44.086815 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:44.086888 (+ 73us) log.cc:844] Serialized 47 byte log entry
0112 02:52:44.086943 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.086962 (+ 19us) write_op.cc:454] Released schema lock
0112 02:52:44.086968 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 02:52:44.271295 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:44.271443 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:44.271452 (+ 9us) write_op.cc:435] Acquired schema lock
0112 02:52:44.271456 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:44.271585 (+ 129us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:44.271599 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:44.271603 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:44.271664 (+ 61us) tablet.cc:698] Row locks acquired
0112 02:52:44.271666 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:44.271699 (+ 33us) write_op.cc:270] Start()
0112 02:52:44.271729 (+ 30us) write_op.cc:276] Timestamp: P: 1736650364271677 usec, L: 0
0112 02:52:44.271733 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:44.271879 (+ 146us) log.cc:844] Serialized 356 byte log entry
0112 02:52:44.275921 (+ 4042us) op_driver.cc:464] REPLICATION: finished
0112 02:52:44.276165 (+ 244us) write_op.cc:301] APPLY: starting
0112 02:52:44.276183 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:44.276234 (+ 51us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:44.276271 (+ 37us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:44.276333 (+ 62us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:44.276388 (+ 55us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:44.276399 (+ 11us) write_op.cc:312] APPLY: finished
0112 02:52:44.276479 (+ 80us) log.cc:844] Serialized 23 byte log entry
0112 02:52:44.276552 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.276573 (+ 21us) write_op.cc:454] Released schema lock
0112 02:52:44.276581 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 02:52:44.764802 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:44.764898 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:44.764902 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:44.764904 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:44.764973 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:44.764981 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:44.764985 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:44.765047 (+ 62us) tablet.cc:698] Row locks acquired
0112 02:52:44.765050 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:44.765075 (+ 25us) write_op.cc:270] Start()
0112 02:52:44.765097 (+ 22us) write_op.cc:276] Timestamp: P: 1736650364765059 usec, L: 0
0112 02:52:44.765100 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:44.765267 (+ 167us) log.cc:844] Serialized 356 byte log entry
0112 02:52:44.769673 (+ 4406us) op_driver.cc:464] REPLICATION: finished
0112 02:52:44.770047 (+ 374us) write_op.cc:301] APPLY: starting
0112 02:52:44.770057 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:44.770082 (+ 25us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:44.770107 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:44.770142 (+ 35us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:44.770165 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:44.770171 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:44.770226 (+ 55us) log.cc:844] Serialized 23 byte log entry
0112 02:52:44.770262 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.770274 (+ 12us) write_op.cc:454] Released schema lock
0112 02:52:44.770279 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:44.770337 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.770341 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:44.805497 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:44.805570 (+ 73us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:44.805575 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:44.805577 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:44.805644 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:44.805651 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:44.805655 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:44.805694 (+ 39us) tablet.cc:698] Row locks acquired
0112 02:52:44.805696 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:44.805719 (+ 23us) write_op.cc:270] Start()
0112 02:52:44.805739 (+ 20us) write_op.cc:276] Timestamp: P: 1736650364805695 usec, L: 0
0112 02:52:44.805742 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:44.810064 (+ 4322us) log.cc:844] Serialized 356 byte log entry
0112 02:52:44.895393 (+ 85329us) op_driver.cc:464] REPLICATION: finished
0112 02:52:44.895989 (+ 596us) write_op.cc:301] APPLY: starting
0112 02:52:44.896005 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:44.896041 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:44.896064 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:44.896111 (+ 47us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:44.896139 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:44.896146 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:44.896208 (+ 62us) log.cc:844] Serialized 23 byte log entry
0112 02:52:44.896264 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.896281 (+ 17us) write_op.cc:454] Released schema lock
0112 02:52:44.896286 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:44.896417 (+ 131us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.896421 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:44.946968 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:44.947048 (+ 80us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:44.947052 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:44.947053 (+ 1us) tablet.cc:659] Decoding operations
0112 02:52:44.947112 (+ 59us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:44.947118 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:44.947122 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:44.947157 (+ 35us) tablet.cc:698] Row locks acquired
0112 02:52:44.947159 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:44.947180 (+ 21us) write_op.cc:270] Start()
0112 02:52:44.947220 (+ 40us) write_op.cc:276] Timestamp: P: 1736650364947128 usec, L: 0
0112 02:52:44.947225 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:44.947337 (+ 112us) log.cc:844] Serialized 356 byte log entry
0112 02:52:44.956778 (+ 9441us) op_driver.cc:464] REPLICATION: finished
0112 02:52:44.961638 (+ 4860us) write_op.cc:301] APPLY: starting
0112 02:52:44.961649 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:44.961673 (+ 24us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:44.962879 (+ 1206us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:44.962924 (+ 45us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:44.962959 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:44.962970 (+ 11us) write_op.cc:312] APPLY: finished
0112 02:52:44.965134 (+ 2164us) log.cc:844] Serialized 23 byte log entry
0112 02:52:44.965173 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.965186 (+ 13us) write_op.cc:454] Released schema lock
0112 02:52:44.965191 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":1595,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":94,"apply.run_cpu_time_us":461,"apply.run_wall_time_us":461,"num_ops":5,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":823,"prepare.run_wall_time_us":823,"raft.queue_time_us":1068,"raft.run_cpu_time_us":348,"raft.run_wall_time_us":349,"replication_time_us":4067}],["op",{"apply.queue_time_us":185,"apply.run_cpu_time_us":479,"apply.run_wall_time_us":479,"num_ops":1,"prepare.queue_time_us":766,"prepare.run_cpu_time_us":695,"prepare.run_wall_time_us":696,"raft.queue_time_us":1081,"raft.run_cpu_time_us":430,"raft.run_wall_time_us":430,"replication_time_us":4154}],["op",{"apply.queue_time_us":331,"apply.run_cpu_time_us":287,"apply.run_wall_time_us":287,"num_ops":1,"prepare.queue_time_us":12521,"prepare.run_cpu_time_us":523,"prepare.run_wall_time_us":523,"replication_time_us":4551}],["op",{"apply.queue_time_us":543,"apply.run_cpu_time_us":359,"apply.run_wall_time_us":422,"num_ops":1,"prepare.queue_time_us":3490,"prepare.run_cpu_time_us":486,"prepare.run_wall_time_us":4637,"replication_time_us":89622}],["op",{"apply.queue_time_us":4830,"apply.run_cpu_time_us":359,"apply.run_wall_time_us":3609,"num_ops":1,"prepare.queue_time_us":1502,"prepare.run_cpu_time_us":406,"prepare.run_wall_time_us":405,"replication_time_us":9540,"thread_start_us":54947,"threads_started":1}]]}
I20250112 02:52:45.128844 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_93133ac10ebe44d58e69c21d228c1c5e" } modify_external_catalogs: true
I20250112 02:52:45.129040 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_93133ac10ebe44d58e69c21d228c1c5e" } modify_external_catalogs: true
I20250112 02:52:45.131213 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ffdc237f8640f0b378809f93fc6292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:45.131421 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d411f906b4d4eb68343bdcc493e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:45.131618 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 13578542e9e741e8817e0ee63b10f1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:45.131806 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d662e0ce13f447595e5f92b94de3018 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:44 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:45.132110 453 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:37987 (request call id 259) took 1404 ms (client timeout 1000 ms). Trace:
I20250112 02:52:45.132485 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 058a9ca089cd4967887b0a1127c7d03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:45.132189 453 rpcz_store.cc:269] 0112 02:52:43.727157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:43.820773 (+ 93616us) service_pool.cc:225] Handling call
0112 02:52:45.132065 (+1311292us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:44.765788 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:44.765870 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:44.765873 (+ 3us) write_op.cc:435] Acquired schema lock
0112 02:52:44.765876 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:44.765938 (+ 62us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:44.765944 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:44.765948 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:44.765984 (+ 36us) tablet.cc:698] Row locks acquired
0112 02:52:44.765986 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:44.766007 (+ 21us) write_op.cc:270] Start()
0112 02:52:44.766050 (+ 43us) write_op.cc:276] Timestamp: P: 1736650364765992 usec, L: 0
0112 02:52:44.766053 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:44.766192 (+ 139us) log.cc:844] Serialized 356 byte log entry
0112 02:52:44.795942 (+ 29750us) op_driver.cc:464] REPLICATION: finished
0112 02:52:44.796050 (+ 108us) write_op.cc:301] APPLY: starting
0112 02:52:44.796063 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:44.796110 (+ 47us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:44.796171 (+ 61us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:44.796224 (+ 53us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:44.796239 (+ 15us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:44.796246 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:44.796308 (+ 62us) log.cc:844] Serialized 23 byte log entry
0112 02:52:44.796351 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.796363 (+ 12us) write_op.cc:454] Released schema lock
0112 02:52:44.796368 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:44.816385 (+ 20017us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.816391 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":44837,"reported_tablets":191,"child_traces":[["op",{"apply.queue_time_us":26,"apply.run_cpu_time_us":389,"apply.run_wall_time_us":20322,"num_ops":1,"prepare.queue_time_us":5151,"prepare.run_cpu_time_us":458,"prepare.run_wall_time_us":456,"replication_time_us":29844,"spinlock_wait_cycles":40576}]]}
W20250112 02:52:45.135398 455 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:58959 (request call id 245) took 1301 ms (client timeout 1000 ms). Trace:
I20250112 02:52:45.135756 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d662e0ce13f447595e5f92b94de3018 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:44 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:45.135960 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 2361bc6bd8164c2daa37be6b0845f4bd with delete_type TABLET_DATA_DELETED (replaced by tablet 154ecbf4175a42a7ad765780c0f902a5) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:45.136005 2179 leader_election.cc:290] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:45.136147 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc67986a45ed414296e68efac8e48fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.136945 3109 log.cc:1198] T a3ffdc237f8640f0b378809f93fc6292 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/a3ffdc237f8640f0b378809f93fc6292
W20250112 02:52:45.137269 714 leader_election.cc:343] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 896908b2726c40f9aabdfc9a6b3283f7
I20250112 02:52:45.137850 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3d291ca3494f05ac23c7fe293e0bd9" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:45.138020 1052 raft_consensus.cc:3054] T 2d3d291ca3494f05ac23c7fe293e0bd9 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.138752 3133 raft_consensus.cc:491] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 02:52:45.138288 455 rpcz_store.cc:269] 0112 02:52:43.833582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.035298 (+201716us) service_pool.cc:225] Handling call
0112 02:52:45.135334 (+1100036us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:44.792336 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:44.792445 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:44.792451 (+ 6us) write_op.cc:435] Acquired schema lock
0112 02:52:44.792467 (+ 16us) tablet.cc:659] Decoding operations
0112 02:52:44.792574 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:44.792582 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:44.792587 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:44.792665 (+ 78us) tablet.cc:698] Row locks acquired
0112 02:52:44.792668 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:44.792695 (+ 27us) write_op.cc:270] Start()
0112 02:52:44.792741 (+ 46us) write_op.cc:276] Timestamp: P: 1736650364792672 usec, L: 0
0112 02:52:44.792744 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:44.792879 (+ 135us) log.cc:844] Serialized 901 byte log entry
0112 02:52:44.821742 (+ 28863us) op_driver.cc:464] REPLICATION: finished
0112 02:52:44.822049 (+ 307us) write_op.cc:301] APPLY: starting
0112 02:52:44.822062 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:44.822095 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:44.822127 (+ 32us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:44.822180 (+ 53us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:44.822207 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:44.822214 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:44.822266 (+ 52us) log.cc:844] Serialized 29 byte log entry
0112 02:52:44.822351 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.822366 (+ 15us) write_op.cc:454] Released schema lock
0112 02:52:44.822371 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:44.822443 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.822458 (+ 15us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1691,"reported_tablets":172,"child_traces":[["op",{"apply.queue_time_us":263,"apply.run_cpu_time_us":373,"apply.run_wall_time_us":373,"num_ops":2,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":597,"prepare.run_wall_time_us":597,"replication_time_us":28973}]]}
I20250112 02:52:45.138904 2451 leader_election.cc:290] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:45.135479 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_fed76a4c2ec74241b1c63266f20f6873"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:45.139911 455 master_service.cc:946] ReplaceTablet: received request to replace tablet be856e4b913e40d58a3f3ba54698cc5f from {username='slave'} at 127.0.0.1:40610
W20250112 02:52:45.140105 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fed76a4c2ec74241b1c63266f20f6873 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.143404 2787 raft_consensus.cc:491] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.143586 2787 raft_consensus.cc:513] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.144341 2787 leader_election.cc:290] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:45.137856 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_96d1a09a902340ff8e05efb75faf3854" } new_table_name: "default.Table_b0b0d76c3b4a428f9ffd93ebb6e96770" modify_external_catalogs: true
I20250112 02:52:45.146291 2977 raft_consensus.cc:513] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.146948 2977 leader_election.cc:290] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:45.149364 454 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40610 (request call id 220) took 1124 ms. Trace:
I20250112 02:52:45.149470 454 rpcz_store.cc:276] 0112 02:52:44.025296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.129284 (+1103988us) service_pool.cc:225] Handling call
0112 02:52:45.140200 (+ 10916us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:52:45.140595 (+ 395us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:52:45.149075 (+ 8480us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:52:45.149205 (+ 130us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:52:45.149302 (+ 97us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:45.141329 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:45.141416 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:45.141420 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:45.141422 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:45.141481 (+ 59us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:45.141488 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:45.141492 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:45.141541 (+ 49us) tablet.cc:698] Row locks acquired
0112 02:52:45.141543 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:45.141563 (+ 20us) write_op.cc:270] Start()
0112 02:52:45.141599 (+ 36us) write_op.cc:276] Timestamp: P: 1736650365141473 usec, L: 0
0112 02:52:45.141602 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:45.141723 (+ 121us) log.cc:844] Serialized 433 byte log entry
0112 02:52:45.145865 (+ 4142us) op_driver.cc:464] REPLICATION: finished
0112 02:52:45.146351 (+ 486us) write_op.cc:301] APPLY: starting
0112 02:52:45.146361 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:45.146398 (+ 37us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:45.146426 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:45.146514 (+ 88us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:45.146543 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:52:45.146549 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:45.146595 (+ 46us) log.cc:844] Serialized 29 byte log entry
0112 02:52:45.146631 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.146644 (+ 13us) write_op.cc:454] Released schema lock
0112 02:52:45.146649 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:45.146708 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.146712 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":455,"apply.run_cpu_time_us":351,"apply.run_wall_time_us":351,"num_ops":2,"prepare.queue_time_us":370,"prepare.run_cpu_time_us":472,"prepare.run_wall_time_us":472,"raft.queue_time_us":270,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":3,"replication_time_us":4249}]]}
I20250112 02:52:45.150799 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_c3e55b9696194ba58cbc9f1bb391b27d" } modify_external_catalogs: true
I20250112 02:52:45.150961 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_c3e55b9696194ba58cbc9f1bb391b27d" } modify_external_catalogs: true
W20250112 02:52:45.138197 446 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:37987 (request call id 258) took 2483 ms (client timeout 1000 ms). Trace:
I20250112 02:52:45.151859 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbef42e11a843b18ed582ab4d04a92a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:45.152050 917 raft_consensus.cc:3054] T 9bbef42e11a843b18ed582ab4d04a92a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:45.151335 446 rpcz_store.cc:269] 0112 02:52:42.654235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:42.915387 (+261152us) service_pool.cc:225] Handling call
0112 02:52:45.138169 (+2222782us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:43.613707 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:43.613831 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:43.613839 (+ 8us) write_op.cc:435] Acquired schema lock
0112 02:52:43.613842 (+ 3us) tablet.cc:659] Decoding operations
0112 02:52:43.614051 (+ 209us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:43.614066 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:43.614071 (+ 5us) tablet.cc:682] Acquiring locks for 3 operations
0112 02:52:43.614173 (+ 102us) tablet.cc:698] Row locks acquired
0112 02:52:43.614176 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:43.614213 (+ 37us) write_op.cc:270] Start()
0112 02:52:43.614247 (+ 34us) write_op.cc:276] Timestamp: P: 1736650363614083 usec, L: 0
0112 02:52:43.614251 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 02:52:43.614485 (+ 234us) log.cc:844] Serialized 1275 byte log entry
0112 02:52:43.745293 (+130808us) op_driver.cc:464] REPLICATION: finished
0112 02:52:43.770687 (+ 25394us) write_op.cc:301] APPLY: starting
0112 02:52:43.770707 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:43.770813 (+ 106us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:43.770848 (+ 35us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:43.770964 (+ 116us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:43.771009 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0112 02:52:43.771019 (+ 10us) write_op.cc:312] APPLY: finished
0112 02:52:43.771136 (+ 117us) log.cc:844] Serialized 35 byte log entry
0112 02:52:43.771225 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:43.771250 (+ 25us) write_op.cc:454] Released schema lock
0112 02:52:43.771261 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:43.799597 (+ 28336us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:43.799603 (+ 6us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:43.780926 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:43.781041 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:43.781048 (+ 7us) write_op.cc:435] Acquired schema lock
0112 02:52:43.781052 (+ 4us) tablet.cc:659] Decoding operations
0112 02:52:43.781149 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:43.781159 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:43.781165 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:43.781249 (+ 84us) tablet.cc:698] Row locks acquired
0112 02:52:43.781252 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:43.781285 (+ 33us) write_op.cc:270] Start()
0112 02:52:43.781316 (+ 31us) write_op.cc:276] Timestamp: P: 1736650363781259 usec, L: 0
0112 02:52:43.781321 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:43.781473 (+ 152us) log.cc:844] Serialized 356 byte log entry
0112 02:52:43.854823 (+ 73350us) op_driver.cc:464] REPLICATION: finished
0112 02:52:43.854941 (+ 118us) write_op.cc:301] APPLY: starting
0112 02:52:43.854953 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:43.854982 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:43.855015 (+ 33us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:43.855060 (+ 45us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:43.855085 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:43.855092 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:43.855149 (+ 57us) log.cc:844] Serialized 23 byte log entry
0112 02:52:43.855351 (+ 202us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:43.855365 (+ 14us) write_op.cc:454] Released schema lock
0112 02:52:43.855371 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:43.855521 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:43.855526 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:52:44.765351 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:44.765422 (+ 71us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:44.765426 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:44.765428 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:44.765494 (+ 66us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:44.765501 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:44.765504 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:52:44.765544 (+ 40us) tablet.cc:698] Row locks acquired
0112 02:52:44.765547 (+ 3us) write_op.cc:260] PREPARE: finished
0112 02:52:44.765581 (+ 34us) write_op.cc:270] Start()
0112 02:52:44.765603 (+ 22us) write_op.cc:276] Timestamp: P: 1736650364765564 usec, L: 0
0112 02:52:44.765606 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:44.765718 (+ 112us) log.cc:844] Serialized 356 byte log entry
0112 02:52:44.769723 (+ 4005us) op_driver.cc:464] REPLICATION: finished
0112 02:52:44.770403 (+ 680us) write_op.cc:301] APPLY: starting
0112 02:52:44.770429 (+ 26us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:44.770452 (+ 23us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:44.770468 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:44.770498 (+ 30us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:44.770520 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:44.770526 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:44.770569 (+ 43us) log.cc:844] Serialized 23 byte log entry
0112 02:52:44.770613 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:44.770624 (+ 11us) write_op.cc:454] Released schema lock
0112 02:52:44.770630 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":14471,"reported_tablets":192,"child_traces":[["op",{"apply.queue_time_us":25326,"apply.run_cpu_time_us":668,"apply.run_wall_time_us":28904,"num_ops":3,"prepare.queue_time_us":367765,"prepare.run_cpu_time_us":6921,"prepare.run_wall_time_us":108132,"raft.queue_time_us":110,"raft.run_cpu_time_us":636,"raft.run_wall_time_us":636,"replication_time_us":131005,"thread_start_us":494462,"threads_started":3,"wal-append.queue_time_us":99740}],["op",{"apply.queue_time_us":82,"apply.run_cpu_time_us":374,"apply.run_wall_time_us":568,"num_ops":1,"prepare.queue_time_us":6127,"prepare.run_cpu_time_us":645,"prepare.run_wall_time_us":1312,"replication_time_us":73476}],["op",{"apply.queue_time_us":652,"apply.run_cpu_time_us":277,"apply.run_wall_time_us":277,"num_ops":1,"prepare.queue_time_us":10691,"prepare.run_cpu_time_us":408,"prepare.run_wall_time_us":406,"replication_time_us":4104,"thread_start_us":21297,"threads_started":1}]]}
I20250112 02:52:45.152431 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430a8652c53e4801a1c72de529c2adcb" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:45.153301 854 leader_election.cc:343] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.154409 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 64b0d58af4cf4adcbeb3ef5070f2982d from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:45.161489 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 4cfe1a1595014c27bbd350cec169d71a
I20250112 02:52:45.162005 452 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40610 (request call id 219) took 1689 ms. Trace:
I20250112 02:52:45.162104 452 rpcz_store.cc:276] 0112 02:52:43.472877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.128714 (+1655837us) service_pool.cc:225] Handling call
0112 02:52:45.129125 (+ 411us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:45.129253 (+ 128us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:52:45.129316 (+ 63us) catalog_manager.cc:2762] Locking tablets
0112 02:52:45.153858 (+ 24542us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:52:45.161334 (+ 7476us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:52:45.161406 (+ 72us) catalog_manager.cc:2810] Committing in-memory state
0112 02:52:45.161441 (+ 35us) catalog_manager.cc:2814] Aborting table tasks
0112 02:52:45.161945 (+ 504us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:45.155350 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:45.155427 (+ 77us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:45.155432 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:45.155434 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:45.155564 (+ 130us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:45.155571 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:45.155574 (+ 3us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:45.155644 (+ 70us) tablet.cc:698] Row locks acquired
0112 02:52:45.155646 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:45.155668 (+ 22us) write_op.cc:270] Start()
0112 02:52:45.155688 (+ 20us) write_op.cc:276] Timestamp: P: 1736650365155575 usec, L: 0
0112 02:52:45.155691 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:45.155798 (+ 107us) log.cc:844] Serialized 725 byte log entry
0112 02:52:45.160441 (+ 4643us) op_driver.cc:464] REPLICATION: finished
0112 02:52:45.161064 (+ 623us) write_op.cc:301] APPLY: starting
0112 02:52:45.161073 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:45.161097 (+ 24us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:45.161120 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:45.161163 (+ 43us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:45.161185 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:45.161192 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:45.161231 (+ 39us) log.cc:844] Serialized 29 byte log entry
0112 02:52:45.161260 (+ 29us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.161270 (+ 10us) write_op.cc:454] Released schema lock
0112 02:52:45.161274 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:45.161330 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.161333 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":599,"apply.run_cpu_time_us":258,"apply.run_wall_time_us":258,"num_ops":2,"prepare.queue_time_us":840,"prepare.run_cpu_time_us":501,"prepare.run_wall_time_us":501,"raft.queue_time_us":1167,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":4737}]]}
I20250112 02:52:45.163015 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_2fcdbb05970f4a098f7ba3e287fa0348" } modify_external_catalogs: true
I20250112 02:52:45.163213 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_2fcdbb05970f4a098f7ba3e287fa0348" } modify_external_catalogs: true
I20250112 02:52:45.165025 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cfe1a1595014c27bbd350cec169d71a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:45.169104 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cfe1a1595014c27bbd350cec169d71a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:45.170650 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430a8652c53e4801a1c72de529c2adcb" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:45.171353 855 leader_election.cc:343] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.171478 855 leader_election.cc:304] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:45.171852 2787 raft_consensus.cc:2743] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:45.170773 3133 raft_consensus.cc:513] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.173707 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_074b886d23664673a2a482c11ba59f15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:45.174194 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet ea33053c62dc4707b759d5c1756ab4bd
I20250112 02:52:45.174532 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 2361bc6bd8164c2daa37be6b0845f4bd with delete_type TABLET_DATA_DELETED (replaced by tablet 154ecbf4175a42a7ad765780c0f902a5) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:45.174605 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet a4328137407843e1879493ef0c56ffc8
I20250112 02:52:45.175019 447 catalog_manager.cc:6596] ReplaceTablet: tablet a4328137407843e1879493ef0c56ffc8 deleted and replaced by tablet a899f8a3a5764d01ab54de18720acce3
I20250112 02:52:45.175158 447 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:40610 (request call id 212) took 2994 ms. Trace:
I20250112 02:52:45.175312 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a7717cdcb8a4f47b11ba7af54b57e3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:43 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:45.175240 447 rpcz_store.cc:276] 0112 02:52:42.180235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.064701 (+2884466us) service_pool.cc:225] Handling call
0112 02:52:45.175107 (+110406us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:45.164979 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:45.165055 (+ 76us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:45.165060 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:45.165062 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:45.165152 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:45.165163 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:45.165168 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:45.165223 (+ 55us) tablet.cc:698] Row locks acquired
0112 02:52:45.165225 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:45.165249 (+ 24us) write_op.cc:270] Start()
0112 02:52:45.165270 (+ 21us) write_op.cc:276] Timestamp: P: 1736650365165153 usec, L: 0
0112 02:52:45.165272 (+ 2us) op_driver.cc:348] REPLICATION: starting
0112 02:52:45.165394 (+ 122us) log.cc:844] Serialized 622 byte log entry
0112 02:52:45.172555 (+ 7161us) op_driver.cc:464] REPLICATION: finished
0112 02:52:45.172767 (+ 212us) write_op.cc:301] APPLY: starting
0112 02:52:45.172777 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:45.172807 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:45.172844 (+ 37us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:45.172932 (+ 88us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:45.172981 (+ 49us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:45.172988 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:45.173037 (+ 49us) log.cc:844] Serialized 29 byte log entry
0112 02:52:45.173087 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.173103 (+ 16us) write_op.cc:454] Released schema lock
0112 02:52:45.173109 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:45.173172 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.173177 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":176,"apply.run_cpu_time_us":396,"apply.run_wall_time_us":396,"num_ops":2,"prepare.queue_time_us":4405,"prepare.run_cpu_time_us":485,"prepare.run_wall_time_us":484,"raft.queue_time_us":1572,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":220,"replication_time_us":7260}]]}
I20250112 02:52:45.175698 451 catalog_manager.cc:6596] ReplaceTablet: tablet ea33053c62dc4707b759d5c1756ab4bd deleted and replaced by tablet aa54d9aa281644479ea38737b9866e87
I20250112 02:52:45.175796 451 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:40610 (request call id 211) took 2997 ms. Trace:
I20250112 02:52:45.175961 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_d0f34244c237425c84c19f0c575857e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.176385 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0f34244c237425c84c19f0c575857e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.176481 746 tablet_service.cc:1514] Processing DeleteTablet for tablet a4328137407843e1879493ef0c56ffc8 with delete_type TABLET_DATA_DELETED (replaced by tablet a899f8a3a5764d01ab54de18720acce3) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:45.176626 749 tablet_service.cc:1514] Processing DeleteTablet for tablet ea33053c62dc4707b759d5c1756ab4bd with delete_type TABLET_DATA_DELETED (replaced by tablet aa54d9aa281644479ea38737b9866e87) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:45.176740 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cfe1a1595014c27bbd350cec169d71a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:45.176820 3138 raft_consensus.cc:491] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.176956 3138 raft_consensus.cc:513] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.175882 451 rpcz_store.cc:276] 0112 02:52:42.177844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.026074 (+2848230us) service_pool.cc:225] Handling call
0112 02:52:45.045741 (+ 19667us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x616000041b54. stack: 00007f3e09869fd6 00007f3e09869d90 00007f3e0884948d 000000000096d544 000000000096d50d 00007f3e18407e1d 00007f3e18407f13 00007f3e15110211 00007f3e1cbe79dd 00007f3e1cd7f0ab 00007f3e13e81b77 00007f3e13e818a5 00007f3e105d130e 00007f3e105d01ee 00007f3e105d6425 00007f3e105d6d4f
0112 02:52:45.175763 (+130022us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:45.165985 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:45.166068 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:45.166072 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:45.166074 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:45.166140 (+ 66us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:45.166147 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:45.166151 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:45.166220 (+ 69us) tablet.cc:698] Row locks acquired
0112 02:52:45.166222 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:45.166243 (+ 21us) write_op.cc:270] Start()
0112 02:52:45.166262 (+ 19us) write_op.cc:276] Timestamp: P: 1736650365166148 usec, L: 0
0112 02:52:45.166265 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:45.166381 (+ 116us) log.cc:844] Serialized 622 byte log entry
0112 02:52:45.172659 (+ 6278us) op_driver.cc:464] REPLICATION: finished
0112 02:52:45.173579 (+ 920us) write_op.cc:301] APPLY: starting
0112 02:52:45.173594 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:45.173649 (+ 55us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:45.173691 (+ 42us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:45.173763 (+ 72us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:45.173812 (+ 49us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:52:45.173818 (+ 6us) write_op.cc:312] APPLY: finished
0112 02:52:45.173883 (+ 65us) log.cc:844] Serialized 29 byte log entry
0112 02:52:45.173921 (+ 38us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.173933 (+ 12us) write_op.cc:454] Released schema lock
0112 02:52:45.173938 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:45.174013 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.174017 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":44581632,"child_traces":[["op",{"apply.queue_time_us":890,"apply.run_cpu_time_us":415,"apply.run_wall_time_us":416,"num_ops":2,"prepare.queue_time_us":2837,"prepare.run_cpu_time_us":456,"prepare.run_wall_time_us":456,"raft.queue_time_us":875,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":2,"replication_time_us":6382}]]}
I20250112 02:52:45.177423 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet f52b838214c54ac69ad30dac6e56669a
I20250112 02:52:45.177595 3138 leader_election.cc:290] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:45.177701 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bbef42e11a843b18ed582ab4d04a92a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:45.177672 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_1a098348f5684cebac211f561607c408"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:45.177862 776 raft_consensus.cc:3054] T 9bbef42e11a843b18ed582ab4d04a92a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.177897 450 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40610 (request call id 218) took 1705 ms. Trace:
I20250112 02:52:45.178110 3133 leader_election.cc:290] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:45.178146 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a098348f5684cebac211f561607c408 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.178372 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet d95f98ae7bdf42518bbff55e257fea88
I20250112 02:52:45.179077 452 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:40574 (request call id 4) took 8871 ms. Trace:
I20250112 02:52:45.179180 452 rpcz_store.cc:276] 0112 02:52:36.307862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.178873 (+8871011us) service_pool.cc:225] Handling call
0112 02:52:45.179035 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:52:45.179165 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5079f87652a4ea58b366d1f9448fd4e" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:45.179352 1051 raft_consensus.cc:2463] T c5079f87652a4ea58b366d1f9448fd4e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:45.179533 452 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:56942 (request call id 0) took 2827 ms. Trace:
I20250112 02:52:45.179601 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1b3ee229c445b19405cf4a19cf8003" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:45.179644 452 rpcz_store.cc:276] 0112 02:52:42.352422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.179330 (+2826908us) service_pool.cc:225] Handling call
0112 02:52:45.179504 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:52:45.179735 1053 raft_consensus.cc:2463] T ff1b3ee229c445b19405cf4a19cf8003 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:45.177973 450 rpcz_store.cc:276] 0112 02:52:43.471971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.112735 (+1640764us) service_pool.cc:225] Handling call
0112 02:52:45.113159 (+ 424us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:45.113312 (+ 153us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:52:45.113362 (+ 50us) catalog_manager.cc:2762] Locking tablets
0112 02:52:45.161104 (+ 47742us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:52:45.177285 (+ 16181us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:52:45.177346 (+ 61us) catalog_manager.cc:2810] Committing in-memory state
0112 02:52:45.177379 (+ 33us) catalog_manager.cc:2814] Aborting table tasks
0112 02:52:45.177842 (+ 463us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:45.165486 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:45.165552 (+ 66us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:45.165556 (+ 4us) write_op.cc:435] Acquired schema lock
0112 02:52:45.165558 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:45.165629 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:45.165635 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:45.165639 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:45.165737 (+ 98us) tablet.cc:698] Row locks acquired
0112 02:52:45.165739 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:45.165762 (+ 23us) write_op.cc:270] Start()
0112 02:52:45.165783 (+ 21us) write_op.cc:276] Timestamp: P: 1736650365165666 usec, L: 0
0112 02:52:45.165786 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:45.165902 (+ 116us) log.cc:844] Serialized 725 byte log entry
0112 02:52:45.172607 (+ 6705us) op_driver.cc:464] REPLICATION: finished
0112 02:52:45.173239 (+ 632us) write_op.cc:301] APPLY: starting
0112 02:52:45.173248 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:45.173274 (+ 26us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:45.173291 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:45.173337 (+ 46us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:45.173360 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:45.173367 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:45.173412 (+ 45us) log.cc:844] Serialized 29 byte log entry
0112 02:52:45.173449 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.173460 (+ 11us) write_op.cc:454] Released schema lock
0112 02:52:45.173465 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:45.173520 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.173523 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":589,"apply.run_cpu_time_us":276,"apply.run_wall_time_us":276,"num_ops":2,"prepare.queue_time_us":3835,"prepare.run_cpu_time_us":476,"prepare.run_wall_time_us":474,"raft.queue_time_us":1327,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":6808}]]}
I20250112 02:52:45.180323 760 tablet_service.cc:1514] Processing DeleteTablet for tablet f52b838214c54ac69ad30dac6e56669a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:45.180408 749 tablet_service.cc:1514] Processing DeleteTablet for tablet d95f98ae7bdf42518bbff55e257fea88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:45.181838 890 tablet_service.cc:1514] Processing DeleteTablet for tablet ea33053c62dc4707b759d5c1756ab4bd with delete_type TABLET_DATA_DELETED (replaced by tablet aa54d9aa281644479ea38737b9866e87) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:45.182013 898 tablet_service.cc:1514] Processing DeleteTablet for tablet a4328137407843e1879493ef0c56ffc8 with delete_type TABLET_DATA_DELETED (replaced by tablet a899f8a3a5764d01ab54de18720acce3) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:45.182114 714 leader_election.cc:304] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:45.182174 897 tablet_service.cc:1514] Processing DeleteTablet for tablet f52b838214c54ac69ad30dac6e56669a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:45.182315 889 tablet_service.cc:1514] Processing DeleteTablet for tablet d95f98ae7bdf42518bbff55e257fea88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:45.182365 714 leader_election.cc:304] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:45.183271 3133 raft_consensus.cc:2798] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:45.183367 3133 raft_consensus.cc:491] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:45.183458 3133 raft_consensus.cc:3054] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.183849 3138 raft_consensus.cc:2798] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:45.184201 3138 raft_consensus.cc:491] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:45.184288 3138 raft_consensus.cc:3054] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.185245 2228 consensus_queue.cc:1035] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:45.185601 2228 consensus_queue.cc:1035] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:52:45.189016 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_074b886d23664673a2a482c11ba59f15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.192420 3109 ts_tablet_manager.cc:1939] T a3ffdc237f8640f0b378809f93fc6292 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:45.194926 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_4194fa71a3a8428799fb963cd3282e24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.195354 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4194fa71a3a8428799fb963cd3282e24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.196108 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 64b0d58af4cf4adcbeb3ef5070f2982d
I20250112 02:52:45.196808 446 catalog_manager.cc:6596] ReplaceTablet: tablet 64b0d58af4cf4adcbeb3ef5070f2982d deleted and replaced by tablet 61d7703b15024fe7a35805345d5e81ab
I20250112 02:52:45.197479 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 64b0d58af4cf4adcbeb3ef5070f2982d with delete_type TABLET_DATA_DELETED (replaced by tablet 61d7703b15024fe7a35805345d5e81ab) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:45.197773 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 64b0d58af4cf4adcbeb3ef5070f2982d with delete_type TABLET_DATA_DELETED (replaced by tablet 61d7703b15024fe7a35805345d5e81ab) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:45.205828 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet ea33053c62dc4707b759d5c1756ab4bd with delete_type TABLET_DATA_DELETED (replaced by tablet aa54d9aa281644479ea38737b9866e87) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:45.207911 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet a4328137407843e1879493ef0c56ffc8 with delete_type TABLET_DATA_DELETED (replaced by tablet a899f8a3a5764d01ab54de18720acce3) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:45.208206 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet f52b838214c54ac69ad30dac6e56669a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:45.208520 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet d95f98ae7bdf42518bbff55e257fea88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:45.208810 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 64b0d58af4cf4adcbeb3ef5070f2982d with delete_type TABLET_DATA_DELETED (replaced by tablet 61d7703b15024fe7a35805345d5e81ab) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:45.209746 989 consensus_peers.cc:487] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54215c55966a46e1ab96e352c4e3764e. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:45.210757 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_d829a029e4764d9290c9b90949472c23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.211282 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d829a029e4764d9290c9b90949472c23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.212936 3149 tablet.cc:1721] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:45.223135 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet be856e4b913e40d58a3f3ba54698cc5f
I20250112 02:52:45.223641 455 catalog_manager.cc:6596] ReplaceTablet: tablet be856e4b913e40d58a3f3ba54698cc5f deleted and replaced by tablet 28c58fc22a444dd2a396cd5aa1c59726
I20250112 02:52:45.224035 449 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet aa54d9aa281644479ea38737b9866e87
I20250112 02:52:45.224148 449 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40610 (request call id 217) took 1759 ms. Trace:
I20250112 02:52:45.224220 449 rpcz_store.cc:276] 0112 02:52:43.464951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.110737 (+1645786us) service_pool.cc:225] Handling call
0112 02:52:45.111097 (+ 360us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:52:45.123619 (+ 12522us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:52:45.123682 (+ 63us) catalog_manager.cc:2762] Locking tablets
0112 02:52:45.174719 (+ 51037us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:52:45.223842 (+ 49123us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:52:45.223916 (+ 74us) catalog_manager.cc:2810] Committing in-memory state
0112 02:52:45.223979 (+ 63us) catalog_manager.cc:2814] Aborting table tasks
0112 02:52:45.224100 (+ 121us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:45.181708 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:52:45.181801 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:52:45.181806 (+ 5us) write_op.cc:435] Acquired schema lock
0112 02:52:45.181808 (+ 2us) tablet.cc:659] Decoding operations
0112 02:52:45.181894 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:52:45.181904 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0112 02:52:45.181911 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:52:45.182001 (+ 90us) tablet.cc:698] Row locks acquired
0112 02:52:45.182003 (+ 2us) write_op.cc:260] PREPARE: finished
0112 02:52:45.182032 (+ 29us) write_op.cc:270] Start()
0112 02:52:45.182056 (+ 24us) write_op.cc:276] Timestamp: P: 1736650365181931 usec, L: 0
0112 02:52:45.182059 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:45.182197 (+ 138us) log.cc:844] Serialized 725 byte log entry
0112 02:52:45.222318 (+ 40121us) op_driver.cc:464] REPLICATION: finished
0112 02:52:45.223021 (+ 703us) write_op.cc:301] APPLY: starting
0112 02:52:45.223030 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0112 02:52:45.223060 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0112 02:52:45.223088 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:52:45.223140 (+ 52us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:52:45.223164 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:52:45.223171 (+ 7us) write_op.cc:312] APPLY: finished
0112 02:52:45.223214 (+ 43us) log.cc:844] Serialized 29 byte log entry
0112 02:52:45.223247 (+ 33us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.223259 (+ 12us) write_op.cc:454] Released schema lock
0112 02:52:45.223263 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0112 02:52:45.223316 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:52:45.223319 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":676,"apply.run_cpu_time_us":291,"apply.run_wall_time_us":290,"num_ops":2,"prepare.queue_time_us":6454,"prepare.run_cpu_time_us":539,"prepare.run_wall_time_us":539,"replication_time_us":40245}]]}
I20250112 02:52:45.225143 454 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet a899f8a3a5764d01ab54de18720acce3
I20250112 02:52:45.227504 889 tablet_service.cc:1514] Processing DeleteTablet for tablet be856e4b913e40d58a3f3ba54698cc5f with delete_type TABLET_DATA_DELETED (replaced by tablet 28c58fc22a444dd2a396cd5aa1c59726) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:45.228787 2228 raft_consensus.cc:491] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.228958 2228 raft_consensus.cc:513] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.229585 2228 leader_election.cc:290] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:45.229982 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_ae080827fc044ff5bf531f16ec4cbd0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.230440 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae080827fc044ff5bf531f16ec4cbd0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.231565 754 tablet_service.cc:1514] Processing DeleteTablet for tablet be856e4b913e40d58a3f3ba54698cc5f with delete_type TABLET_DATA_DELETED (replaced by tablet 28c58fc22a444dd2a396cd5aa1c59726) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:45.234138 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13578542e9e741e8817e0ee63b10f1a1" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:45.234366 772 raft_consensus.cc:2463] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:45.242877 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet be856e4b913e40d58a3f3ba54698cc5f with delete_type TABLET_DATA_DELETED (replaced by tablet 28c58fc22a444dd2a396cd5aa1c59726) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:45.243571 989 leader_election.cc:343] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 13578542e9e741e8817e0ee63b10f1a1
I20250112 02:52:45.243913 2977 consensus_queue.cc:1035] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:45.247287 991 leader_election.cc:304] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:45.247942 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d662e0ce13f447595e5f92b94de3018 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:44 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:45.255039 889 tablet_service.cc:1467] Processing CreateTablet for tablet 9fae49da375f4c6a894c1237dd6621fb (DEFAULT_TABLE table=default.Table_9f9aeec4d85146b5bbe19873ed71d792 [id=6f8b055bfe4949148e26781bc224b040]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.256220 897 tablet_service.cc:1467] Processing CreateTablet for tablet 7deb483f2e7c4a1683185db518d171e8 (DEFAULT_TABLE table=default.Table_b32fcf8b32d44d2093696313ffcb2083 [id=80d39266ce544ffb86ed8549aaeeeb6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.257025 898 tablet_service.cc:1467] Processing CreateTablet for tablet 05ccfb3526e14391ba72acd510fb0cad (DEFAULT_TABLE table=default.Table_294f7525beaa446ba48d602cc44838ed [id=720f355f6de6407ab614a1c9ef088e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.257892 890 tablet_service.cc:1467] Processing CreateTablet for tablet e3454657cbd74544b0ae962736e4b7b5 (DEFAULT_TABLE table=default.Table_d5f4d4c431d24cc0ba82a7dcd012368b [id=3059ab365aff4430b1570224188486fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.263295 754 tablet_service.cc:1467] Processing CreateTablet for tablet 9fae49da375f4c6a894c1237dd6621fb (DEFAULT_TABLE table=default.Table_9f9aeec4d85146b5bbe19873ed71d792 [id=6f8b055bfe4949148e26781bc224b040]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.263361 749 tablet_service.cc:1467] Processing CreateTablet for tablet 7deb483f2e7c4a1683185db518d171e8 (DEFAULT_TABLE table=default.Table_b32fcf8b32d44d2093696313ffcb2083 [id=80d39266ce544ffb86ed8549aaeeeb6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.264081 760 tablet_service.cc:1467] Processing CreateTablet for tablet 05ccfb3526e14391ba72acd510fb0cad (DEFAULT_TABLE table=default.Table_294f7525beaa446ba48d602cc44838ed [id=720f355f6de6407ab614a1c9ef088e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.264204 746 tablet_service.cc:1467] Processing CreateTablet for tablet e3454657cbd74544b0ae962736e4b7b5 (DEFAULT_TABLE table=default.Table_d5f4d4c431d24cc0ba82a7dcd012368b [id=3059ab365aff4430b1570224188486fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.266654 2228 raft_consensus.cc:2798] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:45.266752 2228 raft_consensus.cc:491] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:45.266849 2228 raft_consensus.cc:3054] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.267374 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_84d9fd4fb68b4c31a7f438c12f67d042"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.267943 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84d9fd4fb68b4c31a7f438c12f67d042 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.269987 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 7deb483f2e7c4a1683185db518d171e8 (DEFAULT_TABLE table=default.Table_b32fcf8b32d44d2093696313ffcb2083 [id=80d39266ce544ffb86ed8549aaeeeb6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.270259 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 9fae49da375f4c6a894c1237dd6621fb (DEFAULT_TABLE table=default.Table_9f9aeec4d85146b5bbe19873ed71d792 [id=6f8b055bfe4949148e26781bc224b040]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.270342 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7deb483f2e7c4a1683185db518d171e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.270586 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fae49da375f4c6a894c1237dd6621fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.270901 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 05ccfb3526e14391ba72acd510fb0cad (DEFAULT_TABLE table=default.Table_294f7525beaa446ba48d602cc44838ed [id=720f355f6de6407ab614a1c9ef088e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.271195 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ccfb3526e14391ba72acd510fb0cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.273808 2787 raft_consensus.cc:491] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.273931 2787 raft_consensus.cc:513] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.274479 2787 leader_election.cc:290] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:45.274946 1028 tablet_service.cc:1467] Processing CreateTablet for tablet e3454657cbd74544b0ae962736e4b7b5 (DEFAULT_TABLE table=default.Table_d5f4d4c431d24cc0ba82a7dcd012368b [id=3059ab365aff4430b1570224188486fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.275228 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3454657cbd74544b0ae962736e4b7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.277238 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f559dedfb7854cf890bef362c50b4213" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:45.278468 854 leader_election.cc:343] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.278852 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f559dedfb7854cf890bef362c50b4213" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:45.279341 855 leader_election.cc:343] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.279446 855 leader_election.cc:304] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:45.279546 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_83ec35a32e3e47fc9ccdde96b4f221df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:45.279910 2787 raft_consensus.cc:2743] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:45.279951 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83ec35a32e3e47fc9ccdde96b4f221df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.282994 2787 raft_consensus.cc:491] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.283103 2787 raft_consensus.cc:513] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:45.283226 714 consensus_peers.cc:487] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:45.283797 2787 leader_election.cc:290] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:45.284339 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940affdc2c484b7d9fd2de17f9d1cea7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:45.284648 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940affdc2c484b7d9fd2de17f9d1cea7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:45.285218 854 leader_election.cc:343] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:45.285463 855 leader_election.cc:343] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.285552 855 leader_election.cc:304] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:45.285916 2787 raft_consensus.cc:2743] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:45.289244 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_6a2dccba50e94d42b01b4ad9e1ca5a35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.290272 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a2dccba50e94d42b01b4ad9e1ca5a35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.298127 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_045e5b104dc64217aa39c22534007cab" } new_table_name: "default.Table_16821effaf954ac28a375fb933aef7d3" modify_external_catalogs: true
I20250112 02:52:45.301005 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_4934f6022f2441eda88751236a5798be" } new_table_name: "default.Table_ebbae2e0c50f44fd9e379196c2be90af" modify_external_catalogs: true
I20250112 02:52:45.313405 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_e781b72b17d54bf8ba293cd013e5e14d" } new_table_name: "default.Table_50955c16989f4d08bf187803cedb9dad" modify_external_catalogs: true
I20250112 02:52:45.315681 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_c3651f53d9004655b02efca2428b3ead" } new_table_name: "default.Table_1506fd71c54a4312b1c439a2effa4c69" modify_external_catalogs: true
I20250112 02:52:45.322378 910 raft_consensus.cc:2463] T fd81852962fb43449e35ef45dc1daaad P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
W20250112 02:52:45.329094 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5b741e4f6abd4695b3ba1d7a667b6046 consensus state has no leader
I20250112 02:52:45.334908 1052 raft_consensus.cc:2463] T 2d3d291ca3494f05ac23c7fe293e0bd9 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:45.335493 2745 raft_consensus.cc:513] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.336136 2745 leader_election.cc:290] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:45.340400 2787 raft_consensus.cc:491] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.340533 2787 raft_consensus.cc:513] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.341082 2787 leader_election.cc:290] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:45.341974 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4328137407843e1879493ef0c56ffc8" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:45.342492 855 leader_election.cc:343] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.344435 917 raft_consensus.cc:2463] T 9bbef42e11a843b18ed582ab4d04a92a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:45.344820 989 leader_election.cc:304] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:45.345189 2745 raft_consensus.cc:2798] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:45.345320 2745 raft_consensus.cc:695] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:45.345533 2745 consensus_queue.cc:237] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.346926 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4328137407843e1879493ef0c56ffc8" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:45.347337 854 leader_election.cc:343] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.347436 854 leader_election.cc:304] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:45.347841 2787 raft_consensus.cc:2743] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:45.351822 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_f37babaaa3a241778a5a5542b9b832b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.352401 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f37babaaa3a241778a5a5542b9b832b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.354557 714 leader_election.cc:304] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:45.354866 2445 raft_consensus.cc:2798] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:45.354974 2445 raft_consensus.cc:695] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:45.355185 2445 consensus_queue.cc:237] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.357275 2228 raft_consensus.cc:513] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.357887 2228 leader_election.cc:290] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:45.360018 989 leader_election.cc:343] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 13578542e9e741e8817e0ee63b10f1a1
I20250112 02:52:45.364454 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_e8e62bd2fffe4f50849f722dbece2a2c" } modify_external_catalogs: true
I20250112 02:52:45.364615 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_e8e62bd2fffe4f50849f722dbece2a2c" } modify_external_catalogs: true
I20250112 02:52:45.369004 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fae49da375f4c6a894c1237dd6621fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.369555 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7deb483f2e7c4a1683185db518d171e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.369967 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ccfb3526e14391ba72acd510fb0cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.370304 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3454657cbd74544b0ae962736e4b7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.372586 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 61eac6b572ff47c185768bee7490edf7
I20250112 02:52:45.373703 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 61eac6b572ff47c185768bee7490edf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:45.381918 2179 raft_consensus.cc:491] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.382052 2179 raft_consensus.cc:513] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.382638 2179 leader_election.cc:290] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:45.388227 2787 raft_consensus.cc:491] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.388399 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_03e3448d809c4fc792d0bff94a713044" } modify_external_catalogs: true
I20250112 02:52:45.388348 2787 raft_consensus.cc:513] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.388626 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_03e3448d809c4fc792d0bff94a713044" } modify_external_catalogs: true
I20250112 02:52:45.388657 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125b12a2f9164653be79d3e6d8313d36" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:45.389009 2787 leader_election.cc:290] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:45.389273 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13578542e9e741e8817e0ee63b10f1a1" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:45.389415 779 raft_consensus.cc:3054] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.389636 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b455a37448a044679bd2c0646de3d804" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:45.390206 855 leader_election.cc:343] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.390480 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_83177e7dc14645dbb0e5ef573e80e72f" } new_table_name: "default.Table_630d49265f684b0dbef983cb09ea0a14" modify_external_catalogs: true
I20250112 02:52:45.391633 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b455a37448a044679bd2c0646de3d804" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:45.393884 779 raft_consensus.cc:2463] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
W20250112 02:52:45.394294 854 leader_election.cc:343] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.394414 854 leader_election.cc:304] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:45.394938 2787 raft_consensus.cc:2743] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:45.395424 991 leader_election.cc:343] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.395952 991 leader_election.cc:304] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:45.396363 2228 raft_consensus.cc:2798] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:45.396509 2228 raft_consensus.cc:695] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:45.396754 2228 consensus_queue.cc:237] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:45.398152 989 consensus_peers.cc:487] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62f4bd2df0ce45dabc55373f1154d3ac. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:45.399633 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 26577b614e32458e8bc5d6830c8c2e40
I20250112 02:52:45.400710 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 26577b614e32458e8bc5d6830c8c2e40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:45.401153 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 26577b614e32458e8bc5d6830c8c2e40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:45.408685 2179 raft_consensus.cc:491] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.408828 2179 raft_consensus.cc:513] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.409436 2179 leader_election.cc:290] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:45.415109 3133 raft_consensus.cc:513] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.415218 776 raft_consensus.cc:2463] T 9bbef42e11a843b18ed582ab4d04a92a P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:45.416337 3133 leader_election.cc:290] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:45.417728 3138 raft_consensus.cc:513] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.418251 3138 leader_election.cc:290] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:45.419536 3109 tablet_replica.cc:331] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:45.419953 3110 ts_tablet_manager.cc:1918] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:45.420069 3110 log.cc:1198] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/76009bac22ce4136b148b493a004a0f7
I20250112 02:52:45.420365 3110 ts_tablet_manager.cc:1939] T 76009bac22ce4136b148b493a004a0f7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:45.420886 3110 tablet_replica.cc:331] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:45.420894 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 76009bac22ce4136b148b493a004a0f7 (table default.Table_ea881fde8d694f71b5faf60948630d96 [id=ce171e0d83bd436eaad276bedf32030e]) successfully deleted
I20250112 02:52:45.421118 3110 raft_consensus.cc:2238] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
W20250112 02:52:45.421193 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 8a88463991294bd8a335337f8b072c3d because the tablet was not found. No further retry: Not found: Tablet not found: 8a88463991294bd8a335337f8b072c3d
I20250112 02:52:45.421258 3110 raft_consensus.cc:2267] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:45.421761 3110 ts_tablet_manager.cc:1905] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:45.425006 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 61eac6b572ff47c185768bee7490edf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:45.426239 714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 71719.1us
I20250112 02:52:45.426687 1173 tablet_bootstrap.cc:654] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:45.426963 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet a3ffdc237f8640f0b378809f93fc6292 (table default.Table_fc264b6a06584c7ea757f5d18e05e2bc [id=b560a3488e3d4db58abc7d81d6094d86]) successfully deleted
I20250112 02:52:45.429628 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_7ca45e521bed4ddf8d26c9d32651eb34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.430195 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ca45e521bed4ddf8d26c9d32651eb34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.430976 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_4001552c6dc246e18a8b0ad9041267c6" } modify_external_catalogs: true
I20250112 02:52:45.431124 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_4001552c6dc246e18a8b0ad9041267c6" } modify_external_catalogs: true
I20250112 02:52:45.431653 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_7870d93893b74b4b96d1a3809a9015e6" } modify_external_catalogs: true
I20250112 02:52:45.431818 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_7870d93893b74b4b96d1a3809a9015e6" } modify_external_catalogs: true
I20250112 02:52:45.432322 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_2e65b56ef2f9458ea5125a6f045569a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.432763 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e65b56ef2f9458ea5125a6f045569a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:45.435822 416 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu pid 1229: Timed out: CheckLeaks RPC to 127.0.104.60:39987 timed out after 1.000s (SENT)
I20250112 02:52:45.435962 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu with pid 1229
I20250112 02:52:45.438136 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 26577b614e32458e8bc5d6830c8c2e40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:45.438367 3138 raft_consensus.cc:491] T f69b7a0f52304ad9951a07b3ab0372e7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.438551 3138 raft_consensus.cc:513] T f69b7a0f52304ad9951a07b3ab0372e7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.439183 3138 leader_election.cc:290] T f69b7a0f52304ad9951a07b3ab0372e7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:45.441116 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61eebc5f22664712af6e74e82809ca87 consensus state has no leader
I20250112 02:52:45.441519 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 61eac6b572ff47c185768bee7490edf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:45.445619 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_c8d6f1dc529c44efa678ba41a8cde7e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.446089 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8d6f1dc529c44efa678ba41a8cde7e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.446410 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5079f87652a4ea58b366d1f9448fd4e" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:45.446599 1055 raft_consensus.cc:3054] T c5079f87652a4ea58b366d1f9448fd4e P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.449426 1055 raft_consensus.cc:2463] T c5079f87652a4ea58b366d1f9448fd4e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:45.449708 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1b3ee229c445b19405cf4a19cf8003" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:45.449867 1048 raft_consensus.cc:3054] T ff1b3ee229c445b19405cf4a19cf8003 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.453537 1048 raft_consensus.cc:2463] T ff1b3ee229c445b19405cf4a19cf8003 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:45.446555 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_a27a8c5a296a419fba353c1477c47d18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.456017 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a27a8c5a296a419fba353c1477c47d18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.458120 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_c7dcf1defec042dabf093c4b9b0449a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.458571 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7dcf1defec042dabf093c4b9b0449a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.472478 2787 consensus_queue.cc:1035] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:45.449446 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_ade612ff606d477c84a3de161a684cee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.476151 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ade612ff606d477c84a3de161a684cee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.480429 2977 raft_consensus.cc:491] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 02:52:45.480600 714 leader_election.cc:343] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 05000c6a48e0450d8e6685ca864f67f0
I20250112 02:52:45.480584 2977 raft_consensus.cc:513] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.481186 2977 leader_election.cc:290] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:45.481196 714 leader_election.cc:343] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 24ebf85f76f44633a20d97964638c0ac
I20250112 02:52:45.481359 714 leader_election.cc:304] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:45.481580 3133 raft_consensus.cc:2798] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:45.481678 3133 raft_consensus.cc:695] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:45.481992 1181 tablet_bootstrap.cc:492] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:45.481907 3133 consensus_queue.cc:237] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.482112 1181 ts_tablet_manager.cc:1397] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.394s user 0.035s sys 0.030s
I20250112 02:52:45.482784 1181 raft_consensus.cc:357] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.483001 1181 raft_consensus.cc:383] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:45.483047 1181 raft_consensus.cc:738] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:45.483237 1181 consensus_queue.cc:260] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.483764 1181 ts_tablet_manager.cc:1428] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:45.484007 1181 tablet_bootstrap.cc:492] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:45.484625 714 leader_election.cc:304] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:45.485379 3138 raft_consensus.cc:2798] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:45.486084 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69b7a0f52304ad9951a07b3ab0372e7" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:45.486243 917 raft_consensus.cc:2463] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:45.486918 989 leader_election.cc:304] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:45.486771 3167 tablet.cc:1721] T abfaad67d5e34c0a8338dccc0ef05466 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:45.486809 3168 tablet.cc:1721] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20250112 02:52:45.487190 2228 raft_consensus.cc:2798] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:45.487263 2228 raft_consensus.cc:491] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:45.487330 2228 raft_consensus.cc:3054] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.490029 2228 raft_consensus.cc:513] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.490705 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69b7a0f52304ad9951a07b3ab0372e7" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:45.490904 1048 raft_consensus.cc:2388] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 289afc116aa74408aa5bff74e11c50d8 in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
I20250112 02:52:45.491879 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69b7a0f52304ad9951a07b3ab0372e7" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:45.492023 775 raft_consensus.cc:3054] T f69b7a0f52304ad9951a07b3ab0372e7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.491879 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69b7a0f52304ad9951a07b3ab0372e7" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:45.493309 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_98f19112ff3f49d3913d9c9d1ac87100"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.493862 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98f19112ff3f49d3913d9c9d1ac87100 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.494570 1181 tablet_bootstrap.cc:654] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:45.494602 775 raft_consensus.cc:2463] T f69b7a0f52304ad9951a07b3ab0372e7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:45.494959 991 leader_election.cc:304] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:45.495352 2977 raft_consensus.cc:2798] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:45.495573 2977 raft_consensus.cc:695] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:45.495805 2977 consensus_queue.cc:237] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.495924 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_8bb2bb131d6b4262b964926902af01d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.496366 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8bb2bb131d6b4262b964926902af01d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.497228 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69b7a0f52304ad9951a07b3ab0372e7" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:45.497713 917 raft_consensus.cc:3054] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.497869 2228 leader_election.cc:290] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:45.500470 917 raft_consensus.cc:2463] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:45.501674 3111 ts_tablet_manager.cc:1918] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:45.501921 3111 log.cc:1198] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/7e1954e7859f422f9690fe4a9384a8bb
I20250112 02:52:45.509712 1173 tablet_bootstrap.cc:492] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:45.510396 1173 ts_tablet_manager.cc:1397] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.443s user 0.004s sys 0.002s
I20250112 02:52:45.510928 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_24cecfd421cd4047972cc0a6255ef0c0" } new_table_name: "default.Table_ebadf77c864c4883a04b9f62f15d7f49" modify_external_catalogs: true
I20250112 02:52:45.512794 2985 raft_consensus.cc:2743] T 5a3af0ea4e954053ae89814f3e74e6b7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:52:45.514191 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_f952a0479e8d478bbd9484fc5bf041c8" } modify_external_catalogs: true
I20250112 02:52:45.514459 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_f952a0479e8d478bbd9484fc5bf041c8" } modify_external_catalogs: true
I20250112 02:52:45.515861 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_692d9e1bfcfd435e8c420696c1e5aae5" } new_table_name: "default.Table_dcdbe8e16ced4132a8b540781fd01d2e" modify_external_catalogs: true
I20250112 02:52:45.518855 1177 tablet_bootstrap.cc:654] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:45.519965 774 consensus_meta.cc:287] T 9b81aee21510452881948e41d7aba9be P 289afc116aa74408aa5bff74e11c50d8: Time spent flushing consensus metadata: real 0.555s user 0.003s sys 0.000s
I20250112 02:52:45.520642 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125b12a2f9164653be79d3e6d8313d36" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:45.520807 910 raft_consensus.cc:3054] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.523187 910 raft_consensus.cc:2463] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:45.523612 989 leader_election.cc:304] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:45.524295 2228 raft_consensus.cc:2798] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:45.524421 2228 raft_consensus.cc:695] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:45.524653 2228 consensus_queue.cc:237] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.525099 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ccfb3526e14391ba72acd510fb0cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.526506 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fae49da375f4c6a894c1237dd6621fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.535034 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3454657cbd74544b0ae962736e4b7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.536955 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_43f7ba33caf249c4ba386a9199462957"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.537561 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43f7ba33caf249c4ba386a9199462957 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.539713 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_4770176caf32467daca85ee46a034acd" } new_table_name: "default.Table_476e5893d48e49a9a3fa17be765213d9" modify_external_catalogs: true
I20250112 02:52:45.541560 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7deb483f2e7c4a1683185db518d171e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.542557 3000 raft_consensus.cc:491] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.542670 3000 raft_consensus.cc:513] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.543237 3000 leader_election.cc:290] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:45.544257 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcfa2b316ddd48faa0dc2dd8db8690f9" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:45.544507 854 consensus_peers.cc:487] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca6f080e5fd84084b601b646d4e6c9e2. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:52:45.544939 854 leader_election.cc:343] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:45.546727 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 475025us
I20250112 02:52:45.546911 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcfa2b316ddd48faa0dc2dd8db8690f9" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:45.547403 855 leader_election.cc:343] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.547528 855 leader_election.cc:304] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:45.548133 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896908b2726c40f9aabdfc9a6b3283f7" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:45.548309 919 raft_consensus.cc:2463] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:45.549232 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3d291ca3494f05ac23c7fe293e0bd9" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:45.549388 919 raft_consensus.cc:3054] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.551561 3000 raft_consensus.cc:2743] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:45.551946 919 raft_consensus.cc:2463] T 2d3d291ca3494f05ac23c7fe293e0bd9 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:45.555639 3109 raft_consensus.cc:2238] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:45.555887 3109 raft_consensus.cc:2267] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:45.556461 3109 ts_tablet_manager.cc:1905] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:45.560614 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_65dfa36d9f664916bf107d10764f88cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.561174 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65dfa36d9f664916bf107d10764f88cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.570823 3158 consensus_queue.cc:1035] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:45.495335 3176 tablet.cc:1721] T c73f2fa452b84be398476b8aba6c9a08 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:45.575408 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_2c52fead633649dd869763e5aa532be7" } modify_external_catalogs: true
I20250112 02:52:45.575627 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_2c52fead633649dd869763e5aa532be7" } modify_external_catalogs: true
W20250112 02:52:45.581601 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 05000c6a48e0450d8e6685ca864f67f0 consensus state has no leader
I20250112 02:52:45.589110 3177 tablet.cc:1721] T abfaad67d5e34c0a8338dccc0ef05466 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 02:52:45.590202 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0882ea69a90b4ce3abb4f2da6475972f consensus state has no leader
I20250112 02:52:45.592733 3000 raft_consensus.cc:491] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.592864 3000 raft_consensus.cc:513] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.593516 3000 leader_election.cc:290] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:45.593855 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500f0c5c76e54cd8808e6c8611e99d83" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:45.594408 855 leader_election.cc:343] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.594513 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500f0c5c76e54cd8808e6c8611e99d83" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:45.595518 854 leader_election.cc:343] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.595646 854 leader_election.cc:304] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:45.595865 3000 raft_consensus.cc:2743] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:45.596062 854 consensus_peers.cc:487] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 164f76edcf8f4bd498dcd2c25fd7477d. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:45.597740 1173 raft_consensus.cc:357] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.597919 1173 raft_consensus.cc:383] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:45.597968 1173 raft_consensus.cc:738] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:45.598125 1173 consensus_queue.cc:260] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.598629 1173 ts_tablet_manager.cc:1428] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.088s user 0.000s sys 0.000s
I20250112 02:52:45.598896 1173 tablet_bootstrap.cc:492] T 500f0c5c76e54cd8808e6c8611e99d83 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:45.603026 1181 tablet_bootstrap.cc:492] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:45.603209 1181 ts_tablet_manager.cc:1397] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.119s user 0.005s sys 0.003s
I20250112 02:52:45.613409 1181 raft_consensus.cc:357] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.613713 1181 raft_consensus.cc:383] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:45.613823 1181 raft_consensus.cc:738] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:45.614207 1181 consensus_queue.cc:260] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.615846 3177 tablet.cc:1721] T 2c03b5e612d64314b210c11686fdfa2a P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20250112 02:52:45.618316 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_98f19112ff3f49d3913d9c9d1ac87100" } new_table_name: "default.Table_8e48ce56b9fb4e15bfc3b49e8273b872" modify_external_catalogs: true
I20250112 02:52:45.619958 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_bb82d19aab034576ad7a0303219fa410" } modify_external_catalogs: true
I20250112 02:52:45.620162 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_bb82d19aab034576ad7a0303219fa410" } modify_external_catalogs: true
I20250112 02:52:45.622387 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet e4d42f1611de49d6b6e5aaa9be907054
I20250112 02:52:45.623139 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d42f1611de49d6b6e5aaa9be907054 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:45.623907 889 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d42f1611de49d6b6e5aaa9be907054 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:45.626230 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5079f87652a4ea58b366d1f9448fd4e" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:45.626487 917 raft_consensus.cc:2463] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:45.644398 449 master_service.cc:946] ReplaceTablet: received request to replace tablet f60626faf203441ab48c60147ddc3cf8 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:45.646538 3109 ts_tablet_manager.cc:1918] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:45.654642 1173 tablet_bootstrap.cc:654] T 500f0c5c76e54cd8808e6c8611e99d83 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:45.656312 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_8684181f6c4b40eb8948c3687408a1c4" } new_table_name: "default.Table_ac4a10685c414bef86c147c336335ff7" modify_external_catalogs: true
W20250112 02:52:45.659276 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81c56c2cd7fa45f390602adb67681f04 has no consensus state
I20250112 02:52:45.659826 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 09d06e2e8c13464db8cf801a292bdc99
I20250112 02:52:45.660565 2745 raft_consensus.cc:491] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.660722 2745 raft_consensus.cc:513] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.660954 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_21743df183e844c9a6ca2031b2bc81bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:45.661648 2745 leader_election.cc:290] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:45.663493 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_bfcd8a99635e43a29c4a91082cee18f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:45.668084 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_ef4fac0bfa544b4da7abc353a32d4e6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.676002 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef4fac0bfa544b4da7abc353a32d4e6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.677424 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_35eb917609ca4728bf4823fb912a9680"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.677816 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35eb917609ca4728bf4823fb912a9680 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.677798 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 09d06e2e8c13464db8cf801a292bdc99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:45.679080 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 09d06e2e8c13464db8cf801a292bdc99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:45.679350 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bfcd8a99635e43a29c4a91082cee18f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.680627 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 7b9c845571f44c8eb8753090d63df45f from {username='slave'} at 127.0.0.1:40610
W20250112 02:52:45.682096 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21743df183e844c9a6ca2031b2bc81bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.661895 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c29a9ca43e034b43a00207e5847bbee3" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:45.684368 3109 log.cc:1198] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/b124c15736084e6d8f69b4fb7c41cf39
I20250112 02:52:45.684724 3109 ts_tablet_manager.cc:1939] T b124c15736084e6d8f69b4fb7c41cf39 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
W20250112 02:52:45.684872 991 leader_election.cc:343] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.685735 3109 tablet_replica.cc:331] T 62f4bd2df0ce45dabc55373f1154d3ac P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:45.691172 2787 consensus_queue.cc:1035] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:45.696656 2787 raft_consensus.cc:491] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.696785 2787 raft_consensus.cc:513] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.697422 2787 leader_election.cc:290] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:45.698670 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet b124c15736084e6d8f69b4fb7c41cf39 (table default.Table_f0fb142b9aab490bb067d3cf64dc47b1 [id=1991c0821ef4478eb0eb9282ab772c0c]) successfully deleted
W20250112 02:52:45.699136 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet c29a9ca43e034b43a00207e5847bbee3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c29a9ca43e034b43a00207e5847bbee3 already in progress: creating tablet
I20250112 02:52:45.700385 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_b47e051ef51c499dbda95bbe0b3cea50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.700911 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b47e051ef51c499dbda95bbe0b3cea50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.704991 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24ebf85f76f44633a20d97964638c0ac" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:45.705168 918 raft_consensus.cc:2463] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:45.706245 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05000c6a48e0450d8e6685ca864f67f0" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:45.706398 918 raft_consensus.cc:2463] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:45.706951 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5079f87652a4ea58b366d1f9448fd4e" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:45.707083 918 raft_consensus.cc:3054] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.709765 918 raft_consensus.cc:2463] T c5079f87652a4ea58b366d1f9448fd4e P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:45.710729 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69b7a0f52304ad9951a07b3ab0372e7" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:45.710961 918 raft_consensus.cc:2388] T f69b7a0f52304ad9951a07b3ab0372e7 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 289afc116aa74408aa5bff74e11c50d8 in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
I20250112 02:52:45.712540 1173 tablet_bootstrap.cc:492] T 500f0c5c76e54cd8808e6c8611e99d83 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:45.712736 1173 ts_tablet_manager.cc:1397] T 500f0c5c76e54cd8808e6c8611e99d83 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.114s user 0.003s sys 0.000s
I20250112 02:52:45.713687 1173 raft_consensus.cc:357] T 500f0c5c76e54cd8808e6c8611e99d83 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.713929 1173 raft_consensus.cc:383] T 500f0c5c76e54cd8808e6c8611e99d83 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:45.714033 1173 raft_consensus.cc:738] T 500f0c5c76e54cd8808e6c8611e99d83 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:45.714290 1173 consensus_queue.cc:260] T 500f0c5c76e54cd8808e6c8611e99d83 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.714876 1173 ts_tablet_manager.cc:1428] T 500f0c5c76e54cd8808e6c8611e99d83 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:45.715207 1173 tablet_bootstrap.cc:492] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:45.726300 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 7b9c845571f44c8eb8753090d63df45f
I20250112 02:52:45.726864 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b9c845571f44c8eb8753090d63df45f with delete_type TABLET_DATA_DELETED (replaced by tablet 1ca98a3f0fac4d2590c2c88aea34281e) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:45.727299 454 catalog_manager.cc:6596] ReplaceTablet: tablet 7b9c845571f44c8eb8753090d63df45f deleted and replaced by tablet 1ca98a3f0fac4d2590c2c88aea34281e
I20250112 02:52:45.727367 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b9c845571f44c8eb8753090d63df45f with delete_type TABLET_DATA_DELETED (replaced by tablet 1ca98a3f0fac4d2590c2c88aea34281e) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:45.731532 3109 raft_consensus.cc:2238] T 62f4bd2df0ce45dabc55373f1154d3ac P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:45.731683 3109 raft_consensus.cc:2267] T 62f4bd2df0ce45dabc55373f1154d3ac P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:45.732690 3109 ts_tablet_manager.cc:1905] T 62f4bd2df0ce45dabc55373f1154d3ac P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:45.694558 3138 raft_consensus.cc:695] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:45.700434 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f950c604609042da96e17398f42c1929" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:45.735684 3138 consensus_queue.cc:237] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:45.736014 855 leader_election.cc:343] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.703783 713 leader_election.cc:304] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:45.736891 3175 raft_consensus.cc:2798] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:45.737010 3175 raft_consensus.cc:491] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:45.737123 3175 raft_consensus.cc:3054] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.739778 3175 raft_consensus.cc:513] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.740468 3175 leader_election.cc:290] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:45.740876 746 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d42f1611de49d6b6e5aaa9be907054 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:45.725170 1177 tablet_bootstrap.cc:492] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:45.741353 1177 ts_tablet_manager.cc:1397] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.678s user 0.005s sys 0.000s
I20250112 02:52:45.742146 1177 raft_consensus.cc:357] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.742465 1177 raft_consensus.cc:383] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:45.742565 1177 raft_consensus.cc:738] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:45.742825 1177 consensus_queue.cc:260] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.743443 1177 ts_tablet_manager.cc:1428] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:45.743875 1177 tablet_bootstrap.cc:492] T 8d3a4e76b724452c8b238aae0198e297 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:45.745954 2787 raft_consensus.cc:491] T 4a8ad3cda8124845824024cb69ff54ba P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.746127 2787 raft_consensus.cc:513] T 4a8ad3cda8124845824024cb69ff54ba P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.747023 2787 leader_election.cc:290] T 4a8ad3cda8124845824024cb69ff54ba P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:45.747282 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8ad3cda8124845824024cb69ff54ba" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:45.747720 855 leader_election.cc:343] T 4a8ad3cda8124845824024cb69ff54ba P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:45.748934 713 leader_election.cc:343] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: ff1b3ee229c445b19405cf4a19cf8003
I20250112 02:52:45.749794 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_327f33bee2d64c39a26bbcc21fe16736"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.750321 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_327f33bee2d64c39a26bbcc21fe16736 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.751318 2787 raft_consensus.cc:491] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.751428 2787 raft_consensus.cc:513] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.752216 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_a12ca3f6249040c8bc457ee894d7a274"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:45.752342 2787 leader_election.cc:290] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:45.752667 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a12ca3f6249040c8bc457ee894d7a274 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.754153 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_ce6e0afd36bf48c4937196300248bc26" } new_table_name: "default.Table_0c453685abe143aa82c64b441ec582bb" modify_external_catalogs: true
I20250112 02:52:45.756651 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a243d008c17a46668a3d40e015e84f11" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:45.757007 855 leader_election.cc:343] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.757721 713 leader_election.cc:304] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
W20250112 02:52:45.758118 713 consensus_peers.cc:487] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d63d411f906b4d4eb68343bdcc493e05. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:45.758373 713 leader_election.cc:304] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:45.758680 3175 raft_consensus.cc:2798] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:45.758767 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 09d06e2e8c13464db8cf801a292bdc99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:45.758967 3175 raft_consensus.cc:491] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:45.759028 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b9c845571f44c8eb8753090d63df45f with delete_type TABLET_DATA_DELETED (replaced by tablet 1ca98a3f0fac4d2590c2c88aea34281e) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:45.760118 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f950c604609042da96e17398f42c1929" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:45.760507 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a243d008c17a46668a3d40e015e84f11" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:45.760914 854 leader_election.cc:343] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.761052 2787 raft_consensus.cc:491] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.761190 2787 raft_consensus.cc:513] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.761360 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8ad3cda8124845824024cb69ff54ba" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:45.761967 2787 leader_election.cc:290] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:45.762100 2787 raft_consensus.cc:491] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.762213 2787 raft_consensus.cc:513] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.762913 2787 leader_election.cc:290] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:45.763159 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd52261c4a414e7d93a528e2eddfa9db" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:45.763582 855 leader_election.cc:343] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.763891 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d38f5e525440f8a0f7822620933435" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:45.759073 3175 raft_consensus.cc:3054] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:45.764346 855 leader_election.cc:343] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.764545 854 leader_election.cc:304] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:45.764918 854 leader_election.cc:343] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.765049 854 leader_election.cc:304] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:45.765439 854 leader_election.cc:343] T 4a8ad3cda8124845824024cb69ff54ba P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.765561 854 leader_election.cc:304] T 4a8ad3cda8124845824024cb69ff54ba P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:45.765879 3183 raft_consensus.cc:2743] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:45.766130 3183 raft_consensus.cc:2743] T 4a8ad3cda8124845824024cb69ff54ba P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:45.765713 2787 raft_consensus.cc:2743] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:45.766863 3175 raft_consensus.cc:513] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.768039 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d38f5e525440f8a0f7822620933435" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:45.768474 989 leader_election.cc:343] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: c29a9ca43e034b43a00207e5847bbee3
I20250112 02:52:45.768605 989 leader_election.cc:304] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:45.769016 2745 raft_consensus.cc:2743] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:45.769095 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_4179e7e64aa24195973d2abac70761d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:45.769343 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd52261c4a414e7d93a528e2eddfa9db" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:45.769589 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4179e7e64aa24195973d2abac70761d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:45.769837 854 leader_election.cc:343] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.769919 854 leader_election.cc:304] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:45.770088 3183 raft_consensus.cc:2743] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:45.770419 854 leader_election.cc:343] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.770498 854 leader_election.cc:304] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:45.770651 3110 ts_tablet_manager.cc:1918] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:45.770685 3183 raft_consensus.cc:2743] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:45.770901 3110 log.cc:1198] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/cbe30f9860e2492b90e8cedf8ff9daba
I20250112 02:52:45.771088 3133 raft_consensus.cc:2798] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:45.771234 3133 raft_consensus.cc:491] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:45.771284 3110 ts_tablet_manager.cc:1939] T cbe30f9860e2492b90e8cedf8ff9daba P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:45.771347 3133 raft_consensus.cc:3054] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.772265 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet cbe30f9860e2492b90e8cedf8ff9daba (table default.Table_75f163a2f4aa48df95c9bafd3b311f4b [id=e6546a8535524b5996680a4e5451032f]) successfully deleted
W20250112 02:52:45.772776 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 8429a3d8636d4e2ca4c1191f85b9266b because the tablet was not found. No further retry: Not found: Tablet not found: 8429a3d8636d4e2ca4c1191f85b9266b
W20250112 02:52:45.773520 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 681a930541a84f0197ecafe670278e70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 681a930541a84f0197ecafe670278e70 already in progress: creating tablet
W20250112 02:52:45.774061 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet d0cb54003a83472cb9c57ace3e65b0be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0cb54003a83472cb9c57ace3e65b0be already in progress: creating tablet
I20250112 02:52:45.774255 3133 raft_consensus.cc:513] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:45.774627 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 3e378a6a9b4a455c9f5e1d1ec0fd6ca0 because the tablet was not found. No further retry: Not found: Tablet not found: 3e378a6a9b4a455c9f5e1d1ec0fd6ca0
I20250112 02:52:45.774858 3133 leader_election.cc:290] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:45.775200 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 058a9ca089cd4967887b0a1127c7d03f because the tablet was not found. No further retry: Not found: Tablet not found: 058a9ca089cd4967887b0a1127c7d03f
I20250112 02:52:45.775611 3110 tablet_replica.cc:331] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
W20250112 02:52:45.775998 713 leader_election.cc:343] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: ff1b3ee229c445b19405cf4a19cf8003
I20250112 02:52:45.776185 3110 raft_consensus.cc:2238] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:45.776291 713 leader_election.cc:304] T f69b7a0f52304ad9951a07b3ab0372e7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:45.776512 3173 raft_consensus.cc:2743] T f69b7a0f52304ad9951a07b3ab0372e7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:45.776872 3110 raft_consensus.cc:2267] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:45.777835 3110 ts_tablet_manager.cc:1905] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:45.781450 3167 tablet.cc:1721] T 3f5b418c17084420b867498fda79fb0a P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 02:52:45.786216 1173 tablet_bootstrap.cc:654] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:45.794883 1177 tablet_bootstrap.cc:654] T 8d3a4e76b724452c8b238aae0198e297 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:45.812011 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_6730101812ca4bca8e85617d952cc549" } new_table_name: "default.Table_e82fa0f576134ad087c035f9516863a3" modify_external_catalogs: true
I20250112 02:52:45.813567 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_745092ddecfa47d3a2772ff78e6eab05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:45.814096 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_745092ddecfa47d3a2772ff78e6eab05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:45.816567 3175 leader_election.cc:290] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:45.826642 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_d829a029e4764d9290c9b90949472c23" } new_table_name: "default.Table_fe00ca2755c7455f8b6c4629368d8a26" modify_external_catalogs: true
I20250112 02:52:45.829392 1173 tablet_bootstrap.cc:492] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:45.829577 1173 ts_tablet_manager.cc:1397] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.114s user 0.003s sys 0.000s
I20250112 02:52:45.830287 1173 raft_consensus.cc:357] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.830544 1173 raft_consensus.cc:383] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:45.830641 1173 raft_consensus.cc:738] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:45.830886 1173 consensus_queue.cc:260] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.833709 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896908b2726c40f9aabdfc9a6b3283f7" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:45.780174 1181 ts_tablet_manager.cc:1428] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.177s user 0.006s sys 0.002s
I20250112 02:52:45.833881 911 raft_consensus.cc:3054] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.840652 1181 tablet_bootstrap.cc:492] T f29fb4cc96f64cbeabad085f1007716f P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:45.840907 3138 consensus_queue.cc:1035] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:45.842531 911 raft_consensus.cc:2463] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:45.845466 1181 tablet_bootstrap.cc:654] T f29fb4cc96f64cbeabad085f1007716f P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:45.850150 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24ebf85f76f44633a20d97964638c0ac" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:45.850458 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05000c6a48e0450d8e6685ca864f67f0" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:45.850845 909 raft_consensus.cc:3054] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.850926 919 raft_consensus.cc:3054] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:45.853369 919 raft_consensus.cc:2463] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
W20250112 02:52:45.853490 713 consensus_peers.cc:487] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e1954e7859f422f9690fe4a9384a8bb. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:45.853493 909 raft_consensus.cc:2463] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:45.856781 3207 tablet.cc:1721] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 02:52:45.859688 714 leader_election.cc:343] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 896908b2726c40f9aabdfc9a6b3283f7
I20250112 02:52:45.860860 3109 ts_tablet_manager.cc:1918] T 62f4bd2df0ce45dabc55373f1154d3ac P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:45.861485 1173 ts_tablet_manager.cc:1428] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.032s user 0.001s sys 0.000s
I20250112 02:52:45.862085 3109 log.cc:1198] T 62f4bd2df0ce45dabc55373f1154d3ac P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/62f4bd2df0ce45dabc55373f1154d3ac
I20250112 02:52:45.862211 713 leader_election.cc:304] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:45.862490 3138 raft_consensus.cc:2798] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:45.862548 3109 ts_tablet_manager.cc:1939] T 62f4bd2df0ce45dabc55373f1154d3ac P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:45.862913 1173 tablet_bootstrap.cc:492] T 09900b42e8a443e680cc35c96a1dbbb7 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:45.862634 3138 raft_consensus.cc:695] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:45.863261 3138 consensus_queue.cc:237] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.864135 1177 tablet_bootstrap.cc:492] T 8d3a4e76b724452c8b238aae0198e297 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:45.864295 1177 ts_tablet_manager.cc:1397] T 8d3a4e76b724452c8b238aae0198e297 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.120s user 0.000s sys 0.003s
I20250112 02:52:45.865402 1177 raft_consensus.cc:357] T 8d3a4e76b724452c8b238aae0198e297 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.865658 1177 raft_consensus.cc:383] T 8d3a4e76b724452c8b238aae0198e297 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:45.865765 1177 raft_consensus.cc:738] T 8d3a4e76b724452c8b238aae0198e297 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:45.866065 1177 consensus_queue.cc:260] T 8d3a4e76b724452c8b238aae0198e297 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:45.866695 1177 ts_tablet_manager.cc:1428] T 8d3a4e76b724452c8b238aae0198e297 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 02:52:45.866997 1177 tablet_bootstrap.cc:492] T 058a9ca089cd4967887b0a1127c7d03f P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:45.869310 1177 tablet_bootstrap.cc:654] T 058a9ca089cd4967887b0a1127c7d03f P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:45.870581 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 420) took 1065 ms. Trace:
I20250112 02:52:45.870688 3109 rpcz_store.cc:276] 0112 02:52:44.805408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.805451 (+ 43us) service_pool.cc:225] Handling call
0112 02:52:45.685684 (+880233us) ts_tablet_manager.cc:1145] Deleting tablet 62f4bd2df0ce45dabc55373f1154d3ac
0112 02:52:45.685700 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:45.812232 (+126532us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:45.860742 (+ 48510us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:45.870559 (+ 9817us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":880080}
I20250112 02:52:45.871127 3109 tablet_replica.cc:331] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:45.871342 3109 raft_consensus.cc:2238] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:45.871645 3109 pending_rounds.cc:62] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:45.871703 3109 pending_rounds.cc:69] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319892269862912 op_type: NO_OP noop_request { }
I20250112 02:52:45.871824 3109 raft_consensus.cc:2883] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:45.871913 3109 raft_consensus.cc:2267] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:45.854601 3177 tablet.cc:1721] T 3f5b418c17084420b867498fda79fb0a P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 02:52:45.874325 1173 tablet_bootstrap.cc:654] T 09900b42e8a443e680cc35c96a1dbbb7 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:45.875766 713 leader_election.cc:304] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:45.876143 3106 raft_consensus.cc:2798] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:45.876237 3106 raft_consensus.cc:695] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:45.876412 3106 consensus_queue.cc:237] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:45.861415 714 leader_election.cc:343] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 24ebf85f76f44633a20d97964638c0ac
W20250112 02:52:45.886173 714 leader_election.cc:343] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 05000c6a48e0450d8e6685ca864f67f0
I20250112 02:52:45.893312 3177 tablet.cc:1721] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:45.894636 3111 ts_tablet_manager.cc:1939] T 7e1954e7859f422f9690fe4a9384a8bb P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:45.896569 3110 ts_tablet_manager.cc:1918] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:45.896692 3110 log.cc:1198] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/940affdc2c484b7d9fd2de17f9d1cea7
I20250112 02:52:45.897851 3110 ts_tablet_manager.cc:1939] T 940affdc2c484b7d9fd2de17f9d1cea7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:45.898808 3167 tablet.cc:1721] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:45.899972 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 334) took 1026 ms. Trace:
I20250112 02:52:45.901443 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 7e1954e7859f422f9690fe4a9384a8bb (table default.Table_829049fe22b74e62a87dd1bed65fb3fe [id=2e892eba68234ffca0be4ef3e49cf309]) successfully deleted
I20250112 02:52:45.886590 713 leader_election.cc:304] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters: f1e775296eb746db8b318f17d26820e6
I20250112 02:52:45.890409 3109 ts_tablet_manager.cc:1905] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:45.903126 3174 raft_consensus.cc:2798] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:45.903254 3174 raft_consensus.cc:695] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:45.903548 3174 consensus_queue.cc:237] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.907047 3111 tablet_replica.cc:331] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:45.907557 3111 raft_consensus.cc:2238] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
W20250112 02:52:45.908877 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ff848f9d33e74648a00255c92f599be5 has no consensus state
I20250112 02:52:45.909193 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 37d38f5e525440f8a0f7822620933435 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:45.912468 446 master_service.cc:946] ReplaceTablet: received request to replace tablet cd52261c4a414e7d93a528e2eddfa9db from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:45.900038 3207 tablet.cc:1721] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20250112 02:52:45.919109 3110 rpcz_store.cc:276] 0112 02:52:44.873194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.873655 (+ 461us) service_pool.cc:225] Handling call
0112 02:52:45.775569 (+901914us) ts_tablet_manager.cc:1145] Deleting tablet 940affdc2c484b7d9fd2de17f9d1cea7
0112 02:52:45.775576 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:45.876139 (+100563us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:45.879399 (+ 3260us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:45.899917 (+ 20518us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":901770}
I20250112 02:52:45.916368 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1b3ee229c445b19405cf4a19cf8003 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:45.916488 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet c29a9ca43e034b43a00207e5847bbee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:45.916770 1029 tablet_service.cc:1467] Processing CreateTablet for tablet ff848f9d33e74648a00255c92f599be5 (DEFAULT_TABLE table=default.Table_fe00ca2755c7455f8b6c4629368d8a26 [id=d242943615da4c059601f4047e0f6f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.917104 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 03276edd7bbe42b5a8ca16c5ccf1b337 (DEFAULT_TABLE table=default.Table_074b886d23664673a2a482c11ba59f15 [id=f010fdfe7f1548d89486eed7f6c15303]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.917698 1034 tablet_service.cc:1467] Processing CreateTablet for tablet b768b7534e034bff86045e8a474bf05d (DEFAULT_TABLE table=default.Table_4194fa71a3a8428799fb963cd3282e24 [id=d0f4cc46678f43cba836573b49e845dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.917963 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 648960d4f08f4ce8a4f00aeeb34048f5 (DEFAULT_TABLE table=default.Table_1a098348f5684cebac211f561607c408 [id=148d26dd60bd48efbb72d5770d4c0380]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.921878 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff848f9d33e74648a00255c92f599be5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.922034 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648960d4f08f4ce8a4f00aeeb34048f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.923640 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03276edd7bbe42b5a8ca16c5ccf1b337. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.926833 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b768b7534e034bff86045e8a474bf05d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.926954 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 62f4bd2df0ce45dabc55373f1154d3ac (table default.Table_7921b99e24f84cceb58128f18925ad49 [id=0525f87eccbb4a1d8726d18a00f9a4e4]) successfully deleted
I20250112 02:52:45.928390 3177 tablet.cc:1721] T e3ee5d8a17874d7ba54de51568e7d2e6 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:45.932019 3111 raft_consensus.cc:2267] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:45.932483 3111 ts_tablet_manager.cc:1905] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:45.933413 3110 tablet_replica.cc:331] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:45.933637 3110 raft_consensus.cc:2238] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:45.934170 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 940affdc2c484b7d9fd2de17f9d1cea7 (table default.Table_9afc6941825a45bc8f98bf373ff5f27c [id=c76eb62dc8bf4b97b675787374189057]) successfully deleted
W20250112 02:52:45.938103 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 0882ea69a90b4ce3abb4f2da6475972f (table default.Table_476e5893d48e49a9a3fa17be765213d9 [id=45d7bbac986d4db9a3e0b6f1a88ffa1d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0882ea69a90b4ce3abb4f2da6475972f consensus state has no leader
I20250112 02:52:45.919236 897 tablet_service.cc:1467] Processing CreateTablet for tablet b768b7534e034bff86045e8a474bf05d (DEFAULT_TABLE table=default.Table_4194fa71a3a8428799fb963cd3282e24 [id=d0f4cc46678f43cba836573b49e845dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.939782 3110 raft_consensus.cc:2267] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:45.939966 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b768b7534e034bff86045e8a474bf05d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.914574 889 tablet_service.cc:1467] Processing CreateTablet for tablet ff848f9d33e74648a00255c92f599be5 (DEFAULT_TABLE table=default.Table_fe00ca2755c7455f8b6c4629368d8a26 [id=d242943615da4c059601f4047e0f6f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.943965 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff848f9d33e74648a00255c92f599be5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.944247 3110 ts_tablet_manager.cc:1905] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:45.918897 890 tablet_service.cc:1467] Processing CreateTablet for tablet 03276edd7bbe42b5a8ca16c5ccf1b337 (DEFAULT_TABLE table=default.Table_074b886d23664673a2a482c11ba59f15 [id=f010fdfe7f1548d89486eed7f6c15303]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.947789 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03276edd7bbe42b5a8ca16c5ccf1b337. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.952333 899 tablet_service.cc:1467] Processing CreateTablet for tablet 6151006e155440b7b02ab5e927805101 (DEFAULT_TABLE table=default.Table_18c158c848f1430f9852655c57f10933 [id=71b937181f95465598572c55d10e8a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.952436 898 tablet_service.cc:1467] Processing CreateTablet for tablet 648960d4f08f4ce8a4f00aeeb34048f5 (DEFAULT_TABLE table=default.Table_1a098348f5684cebac211f561607c408 [id=148d26dd60bd48efbb72d5770d4c0380]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.952697 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6151006e155440b7b02ab5e927805101. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.952760 895 tablet_service.cc:1467] Processing CreateTablet for tablet 2709ff38dc3e44628bb46d9df9404ee3 (DEFAULT_TABLE table=default.Table_d0f34244c237425c84c19f0c575857e2 [id=5325ba29fd884b31968ee811da6e086a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.953116 887 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2ac6ca668b4482b5438766668a469b (DEFAULT_TABLE table=default.Table_0be1f83063bb4449a2129e293757be37 [id=6f342319226b48a78a2a0267250f9561]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.953481 891 tablet_service.cc:1467] Processing CreateTablet for tablet 4c03c84db516446aaf903622fff58362 (DEFAULT_TABLE table=default.Table_ae080827fc044ff5bf531f16ec4cbd0f [id=c45ce00ef04c4d35a47d840c5608e2de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.953986 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2709ff38dc3e44628bb46d9df9404ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.953996 746 tablet_service.cc:1467] Processing CreateTablet for tablet ff848f9d33e74648a00255c92f599be5 (DEFAULT_TABLE table=default.Table_fe00ca2755c7455f8b6c4629368d8a26 [id=d242943615da4c059601f4047e0f6f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.954407 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff848f9d33e74648a00255c92f599be5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.954519 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e2ac6ca668b4482b5438766668a469b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.955016 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c03c84db516446aaf903622fff58362. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.955524 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648960d4f08f4ce8a4f00aeeb34048f5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:45.956053 441 connection.cc:537] client connection to 127.0.104.60:39987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:52:45.956225 441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
I20250112 02:52:45.956504 424 master_runner.cc:186] Error getting master registration for 127.0.104.60:39987: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107),
I20250112 02:52:45.956640 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:52:45.956892 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 4c03c84db516446aaf903622fff58362 (DEFAULT_TABLE table=default.Table_ae080827fc044ff5bf531f16ec4cbd0f [id=c45ce00ef04c4d35a47d840c5608e2de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.957300 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2ac6ca668b4482b5438766668a469b (DEFAULT_TABLE table=default.Table_0be1f83063bb4449a2129e293757be37 [id=6f342319226b48a78a2a0267250f9561]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.961268 746 tablet_service.cc:1467] Processing CreateTablet for tablet 03276edd7bbe42b5a8ca16c5ccf1b337 (DEFAULT_TABLE table=default.Table_074b886d23664673a2a482c11ba59f15 [id=f010fdfe7f1548d89486eed7f6c15303]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.961314 760 tablet_service.cc:1467] Processing CreateTablet for tablet b768b7534e034bff86045e8a474bf05d (DEFAULT_TABLE table=default.Table_4194fa71a3a8428799fb963cd3282e24 [id=d0f4cc46678f43cba836573b49e845dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.961696 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b768b7534e034bff86045e8a474bf05d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.966200 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 2709ff38dc3e44628bb46d9df9404ee3 (DEFAULT_TABLE table=default.Table_d0f34244c237425c84c19f0c575857e2 [id=5325ba29fd884b31968ee811da6e086a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.966639 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 6151006e155440b7b02ab5e927805101 (DEFAULT_TABLE table=default.Table_18c158c848f1430f9852655c57f10933 [id=71b937181f95465598572c55d10e8a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:45.967366 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03276edd7bbe42b5a8ca16c5ccf1b337. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.968904 3207 tablet.cc:1721] T 3f5b418c17084420b867498fda79fb0a P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 02:52:45.971818 3000 raft_consensus.cc:491] T 7eb24b3d8a9446e8abd9729462225002 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:45.971959 3000 raft_consensus.cc:513] T 7eb24b3d8a9446e8abd9729462225002 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:45.972528 3000 leader_election.cc:290] T 7eb24b3d8a9446e8abd9729462225002 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:45.972812 3109 ts_tablet_manager.cc:1918] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:45.973233 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb24b3d8a9446e8abd9729462225002" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:45.973381 3109 log.cc:1198] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/7e1954e7859f422f9690fe4a9384a8bb
I20250112 02:52:45.973760 3109 ts_tablet_manager.cc:1939] T 7e1954e7859f422f9690fe4a9384a8bb P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
W20250112 02:52:45.973838 855 leader_election.cc:343] T 7eb24b3d8a9446e8abd9729462225002 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:45.975179 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c03c84db516446aaf903622fff58362. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.975138 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb24b3d8a9446e8abd9729462225002" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:45.979614 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e2ac6ca668b4482b5438766668a469b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.980762 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6151006e155440b7b02ab5e927805101. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:45.984493 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet f60626faf203441ab48c60147ddc3cf8
I20250112 02:52:45.985003 449 catalog_manager.cc:6596] ReplaceTablet: tablet f60626faf203441ab48c60147ddc3cf8 deleted and replaced by tablet 2473488c4890403291e38f287a976851
I20250112 02:52:45.985229 449 master_service.cc:946] ReplaceTablet: received request to replace tablet d2028226498b4b41a95c7c836fe03b4d from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:45.985539 455 catalog_manager.cc:5526] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:45.989528 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 413) took 1184 ms. Trace:
I20250112 02:52:45.991941 3110 ts_tablet_manager.cc:1918] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:45.992237 3109 rpcz_store.cc:276] 0112 02:52:44.804824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.805456 (+ 632us) service_pool.cc:225] Handling call
0112 02:52:45.871077 (+1065621us) ts_tablet_manager.cc:1145] Deleting tablet 7e1954e7859f422f9690fe4a9384a8bb
0112 02:52:45.871089 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:45.906307 (+ 35218us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:45.953559 (+ 47252us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:45.989508 (+ 35949us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1065363}
I20250112 02:52:45.992910 3109 tablet_replica.cc:331] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
W20250112 02:52:45.994732 990 connection.cc:537] client connection to 127.0.104.60:39987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:52:45.994756 2246 connection.cc:537] client connection to 127.0.104.60:39987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 02:52:46.002053 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet f60626faf203441ab48c60147ddc3cf8 with delete_type TABLET_DATA_DELETED (replaced by tablet 2473488c4890403291e38f287a976851) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.002333 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2709ff38dc3e44628bb46d9df9404ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.014781 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 7e1954e7859f422f9690fe4a9384a8bb (table default.Table_829049fe22b74e62a87dd1bed65fb3fe [id=2e892eba68234ffca0be4ef3e49cf309]) successfully deleted
I20250112 02:52:45.994980 1778 master_runner.cc:186] Error getting master registration for 127.0.104.60:39987: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107),
I20250112 02:52:46.017026 1778 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:52:45.994529 1177 tablet_bootstrap.cc:492] T 058a9ca089cd4967887b0a1127c7d03f P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:46.004226 754 tablet_service.cc:1467] Processing CreateTablet for tablet 648960d4f08f4ce8a4f00aeeb34048f5 (DEFAULT_TABLE table=default.Table_1a098348f5684cebac211f561607c408 [id=148d26dd60bd48efbb72d5770d4c0380]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.024915 3167 tablet.cc:1721] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:46.024415 1177 ts_tablet_manager.cc:1397] T 058a9ca089cd4967887b0a1127c7d03f P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.157s user 0.003s sys 0.000s
I20250112 02:52:46.028246 1177 raft_consensus.cc:357] T 058a9ca089cd4967887b0a1127c7d03f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.028914 1177 raft_consensus.cc:383] T 058a9ca089cd4967887b0a1127c7d03f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:46.029024 1177 raft_consensus.cc:738] T 058a9ca089cd4967887b0a1127c7d03f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:46.029269 1177 consensus_queue.cc:260] T 058a9ca089cd4967887b0a1127c7d03f P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.029932 1177 ts_tablet_manager.cc:1428] T 058a9ca089cd4967887b0a1127c7d03f P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:46.005757 760 tablet_service.cc:1467] Processing CreateTablet for tablet 6151006e155440b7b02ab5e927805101 (DEFAULT_TABLE table=default.Table_18c158c848f1430f9852655c57f10933 [id=71b937181f95465598572c55d10e8a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.006115 3133 consensus_queue.cc:1035] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:46.006690 747 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2ac6ca668b4482b5438766668a469b (DEFAULT_TABLE table=default.Table_0be1f83063bb4449a2129e293757be37 [id=6f342319226b48a78a2a0267250f9561]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.009231 749 tablet_service.cc:1467] Processing CreateTablet for tablet 4c03c84db516446aaf903622fff58362 (DEFAULT_TABLE table=default.Table_ae080827fc044ff5bf531f16ec4cbd0f [id=c45ce00ef04c4d35a47d840c5608e2de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.009738 743 tablet_service.cc:1467] Processing CreateTablet for tablet 28c58fc22a444dd2a396cd5aa1c59726 (DEFAULT_TABLE table=default.Table_82731e14956e49edb4d2103794e88fec [id=9a8fe3b96ed34c95814c9c19b8804e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.009923 756 tablet_service.cc:1514] Processing DeleteTablet for tablet f60626faf203441ab48c60147ddc3cf8 with delete_type TABLET_DATA_DELETED (replaced by tablet 2473488c4890403291e38f287a976851) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:46.031239 3109 raft_consensus.cc:2238] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250112 02:52:46.040030 3224 negotiation.cc:337] Failed RPC negotiation. Trace:
0112 02:52:45.970173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:39987 (local address 127.0.0.1:37814)
0112 02:52:45.970652 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0112 02:52:45.970670 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0112 02:52:45.970779 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0112 02:52:46.039752 (+ 68973us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":684,"threads_started":1}
I20250112 02:52:46.040372 3109 raft_consensus.cc:2267] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:46.040361 441 consensus_peers.cc:487] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e -> Peer dd25f03f25df49178a5c53c636ecf304 (127.0.104.60:39987): Couldn't send request to peer dd25f03f25df49178a5c53c636ecf304. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20250112 02:52:46.017503 714 connection.cc:537] client connection to 127.0.104.60:39987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 02:52:46.016793 3175 consensus_queue.cc:1035] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:46.038138 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648960d4f08f4ce8a4f00aeeb34048f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.038944 1177 tablet_bootstrap.cc:492] T 125b12a2f9164653be79d3e6d8313d36 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:46.039144 3167 tablet.cc:1721] T 47a0d939fdb94590a86389f1f4693e9a P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 02:52:46.041672 3226 negotiation.cc:337] Failed RPC negotiation. Trace:
0112 02:52:46.000150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:39987 (local address 127.0.104.3:35143)
0112 02:52:46.022669 (+ 22519us) negotiation.cc:107] Waiting for socket to connect
0112 02:52:46.022687 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0112 02:52:46.022797 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0112 02:52:46.041511 (+ 18714us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":22420,"thread_start_us":22351,"threads_started":1}
I20250112 02:52:46.043129 1177 tablet_bootstrap.cc:654] T 125b12a2f9164653be79d3e6d8313d36 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:46.043588 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c03c84db516446aaf903622fff58362. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.044608 1173 tablet_bootstrap.cc:492] T 09900b42e8a443e680cc35c96a1dbbb7 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:46.044718 1173 ts_tablet_manager.cc:1397] T 09900b42e8a443e680cc35c96a1dbbb7 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.182s user 0.003s sys 0.008s
I20250112 02:52:46.045272 1173 raft_consensus.cc:357] T 09900b42e8a443e680cc35c96a1dbbb7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.045436 1173 raft_consensus.cc:383] T 09900b42e8a443e680cc35c96a1dbbb7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:46.045478 1173 raft_consensus.cc:738] T 09900b42e8a443e680cc35c96a1dbbb7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:46.041226 3109 ts_tablet_manager.cc:1905] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.047719 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e2ac6ca668b4482b5438766668a469b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.049285 3111 ts_tablet_manager.cc:1918] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:46.049373 3111 log.cc:1198] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/82b351c563d749df806ab5d0331a4ef2
I20250112 02:52:46.049716 3111 ts_tablet_manager.cc:1939] T 82b351c563d749df806ab5d0331a4ef2 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:46.050133 3111 tablet_replica.cc:331] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:46.050294 3111 raft_consensus.cc:2238] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:46.050539 3111 raft_consensus.cc:2267] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.051017 3111 ts_tablet_manager.cc:1905] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.051662 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28c58fc22a444dd2a396cd5aa1c59726. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.051617 1173 consensus_queue.cc:260] T 09900b42e8a443e680cc35c96a1dbbb7 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.052047 1173 ts_tablet_manager.cc:1428] T 09900b42e8a443e680cc35c96a1dbbb7 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250112 02:52:46.052282 1173 tablet_bootstrap.cc:492] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:46.001633 891 tablet_service.cc:1514] Processing DeleteTablet for tablet f60626faf203441ab48c60147ddc3cf8 with delete_type TABLET_DATA_DELETED (replaced by tablet 2473488c4890403291e38f287a976851) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.052862 3110 log.cc:1198] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/a8e7882e07784bdda18dc1ffd20963c8
I20250112 02:52:46.053190 3110 ts_tablet_manager.cc:1939] T a8e7882e07784bdda18dc1ffd20963c8 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
W20250112 02:52:46.007558 854 leader_election.cc:343] T 7eb24b3d8a9446e8abd9729462225002 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:46.053522 854 leader_election.cc:304] T 7eb24b3d8a9446e8abd9729462225002 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:46.053788 3183 raft_consensus.cc:2743] T 7eb24b3d8a9446e8abd9729462225002 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:46.054131 854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 46574.4us
W20250112 02:52:46.054505 854 connection.cc:537] client connection to 127.0.104.60:39987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 02:52:46.055828 1173 tablet_bootstrap.cc:654] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:46.015986 3221 tablet.cc:1721] T 47a0d939fdb94590a86389f1f4693e9a P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:46.018747 3213 tablet.cc:1721] T e3ee5d8a17874d7ba54de51568e7d2e6 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:46.020108 3000 raft_consensus.cc:491] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:46.058182 3000 raft_consensus.cc:513] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.058712 3000 leader_election.cc:290] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:46.020924 3207 tablet.cc:1721] T abfaad67d5e34c0a8338dccc0ef05466 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:46.066229 3213 tablet.cc:1721] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:46.068397 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet d2028226498b4b41a95c7c836fe03b4d
I20250112 02:52:46.069062 449 catalog_manager.cc:6596] ReplaceTablet: tablet d2028226498b4b41a95c7c836fe03b4d deleted and replaced by tablet 1b34cef7c8a14dbda7c091dfed46515a
I20250112 02:52:46.069577 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 37d38f5e525440f8a0f7822620933435
I20250112 02:52:46.069716 891 tablet_service.cc:1514] Processing DeleteTablet for tablet d2028226498b4b41a95c7c836fe03b4d with delete_type TABLET_DATA_DELETED (replaced by tablet 1b34cef7c8a14dbda7c091dfed46515a) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.070019 448 catalog_manager.cc:6596] ReplaceTablet: tablet 37d38f5e525440f8a0f7822620933435 deleted and replaced by tablet b1cb48c936844d049a1c832b47986d73
I20250112 02:52:46.070310 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 37d38f5e525440f8a0f7822620933435 with delete_type TABLET_DATA_DELETED (replaced by tablet b1cb48c936844d049a1c832b47986d73) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.070791 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet cd52261c4a414e7d93a528e2eddfa9db
I20250112 02:52:46.071192 446 catalog_manager.cc:6596] ReplaceTablet: tablet cd52261c4a414e7d93a528e2eddfa9db deleted and replaced by tablet a7db121c542f423bbffd2e36c459c3f0
I20250112 02:52:46.071556 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_468eef2886374c71b52269b9a47a4772"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:46.071734 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 82b351c563d749df806ab5d0331a4ef2 (table default.Table_4603068c01544e2e965dca5177e36b40 [id=8502f21e9b61436093db1241cfbb5ec9]) successfully deleted
I20250112 02:52:46.071977 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_c26869f6cbd540fe980dcf207b5c729d" } modify_external_catalogs: true
W20250112 02:52:46.072082 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_468eef2886374c71b52269b9a47a4772 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.072163 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_c26869f6cbd540fe980dcf207b5c729d" } modify_external_catalogs: true
I20250112 02:52:46.073364 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6151006e155440b7b02ab5e927805101. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.073453 891 tablet_service.cc:1514] Processing DeleteTablet for tablet cd52261c4a414e7d93a528e2eddfa9db with delete_type TABLET_DATA_DELETED (replaced by tablet a7db121c542f423bbffd2e36c459c3f0) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.040788 1181 tablet_bootstrap.cc:492] T f29fb4cc96f64cbeabad085f1007716f P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:46.074874 1181 ts_tablet_manager.cc:1397] T f29fb4cc96f64cbeabad085f1007716f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.234s user 0.002s sys 0.014s
I20250112 02:52:46.075654 1181 raft_consensus.cc:357] T f29fb4cc96f64cbeabad085f1007716f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:46.075892 1181 raft_consensus.cc:383] T f29fb4cc96f64cbeabad085f1007716f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:46.076011 1181 raft_consensus.cc:738] T f29fb4cc96f64cbeabad085f1007716f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:46.076295 1181 consensus_queue.cc:260] T f29fb4cc96f64cbeabad085f1007716f P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:46.076978 1181 ts_tablet_manager.cc:1428] T f29fb4cc96f64cbeabad085f1007716f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250112 02:52:46.079035 714 consensus_peers.cc:487] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5804fc4130f84c69832c1022b7bd7fa5. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:46.081369 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3499374810fb412e938325434c35618d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:46.081454 2745 raft_consensus.cc:491] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:46.081586 2745 raft_consensus.cc:513] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.081744 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 37d38f5e525440f8a0f7822620933435 with delete_type TABLET_DATA_DELETED (replaced by tablet b1cb48c936844d049a1c832b47986d73) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.081928 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet cd52261c4a414e7d93a528e2eddfa9db with delete_type TABLET_DATA_DELETED (replaced by tablet a7db121c542f423bbffd2e36c459c3f0) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.082224 2745 leader_election.cc:290] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:46.082440 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet d2028226498b4b41a95c7c836fe03b4d with delete_type TABLET_DATA_DELETED (replaced by tablet 1b34cef7c8a14dbda7c091dfed46515a) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.086303 2745 raft_consensus.cc:491] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:46.086429 2745 raft_consensus.cc:513] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.086882 2745 leader_election.cc:290] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:46.092702 891 tablet_service.cc:1467] Processing CreateTablet for tablet 28c58fc22a444dd2a396cd5aa1c59726 (DEFAULT_TABLE table=default.Table_82731e14956e49edb4d2103794e88fec [id=9a8fe3b96ed34c95814c9c19b8804e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.093217 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 28c58fc22a444dd2a396cd5aa1c59726 (DEFAULT_TABLE table=default.Table_82731e14956e49edb4d2103794e88fec [id=9a8fe3b96ed34c95814c9c19b8804e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.093838 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 12e5e15a95b24794b1f2dd837d548a27 (DEFAULT_TABLE table=default.Table_fed76a4c2ec74241b1c63266f20f6873 [id=849b1709534642e3b466dc6dc5e03374]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.094224 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28c58fc22a444dd2a396cd5aa1c59726. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.094527 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12e5e15a95b24794b1f2dd837d548a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.101722 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3a4e76b724452c8b238aae0198e297" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:46.101900 772 raft_consensus.cc:2463] T 8d3a4e76b724452c8b238aae0198e297 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:46.101876 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f16ec377f01484eae10e63490da143f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:46.102635 991 leader_election.cc:304] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:46.102712 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3499374810fb412e938325434c35618d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:46.102905 991 leader_election.cc:343] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:46.103123 855 leader_election.cc:343] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:46.103554 3159 raft_consensus.cc:2798] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:46.103626 3159 raft_consensus.cc:491] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:46.103682 3159 raft_consensus.cc:3054] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:46.106014 3159 raft_consensus.cc:513] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.106560 3159 leader_election.cc:290] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:46.108109 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 335) took 1234 ms. Trace:
I20250112 02:52:46.108291 3110 rpcz_store.cc:276] 0112 02:52:44.873279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.873823 (+ 544us) service_pool.cc:225] Handling call
0112 02:52:45.933351 (+1059528us) ts_tablet_manager.cc:1145] Deleting tablet a8e7882e07784bdda18dc1ffd20963c8
0112 02:52:45.933360 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:45.972331 (+ 38971us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:45.991853 (+ 19522us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.108054 (+116201us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1059358}
I20250112 02:52:46.108908 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet a8e7882e07784bdda18dc1ffd20963c8 (table default.Table_f3695b00a89b4353903587a6c1418d8c [id=d1781738b8a74f139b09863918cbb21d]) successfully deleted
I20250112 02:52:46.108934 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 336) took 1235 ms. Trace:
I20250112 02:52:46.109052 3110 rpcz_store.cc:276] 0112 02:52:44.873384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.873430 (+ 46us) service_pool.cc:225] Handling call
0112 02:52:46.108777 (+1235347us) ts_tablet_manager.cc:1145] Deleting tablet c29a9ca43e034b43a00207e5847bbee3
0112 02:52:46.108789 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.108892 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1234307}
I20250112 02:52:46.109711 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28c58fc22a444dd2a396cd5aa1c59726. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.111964 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3a4e76b724452c8b238aae0198e297" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:46.112116 776 raft_consensus.cc:3054] T 8d3a4e76b724452c8b238aae0198e297 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:46.114517 776 raft_consensus.cc:2463] T 8d3a4e76b724452c8b238aae0198e297 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:46.115051 991 leader_election.cc:304] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:46.115447 2745 raft_consensus.cc:2798] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:46.115599 2745 raft_consensus.cc:695] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:46.110059 1181 tablet_bootstrap.cc:492] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:46.115870 2745 consensus_queue.cc:237] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.125130 1181 tablet_bootstrap.cc:654] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:46.125991 3110 tablet_replica.cc:331] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
W20250112 02:52:46.126199 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet c29a9ca43e034b43a00207e5847bbee3 because the tablet was not found. No further retry: Not found: Tablet not found: c29a9ca43e034b43a00207e5847bbee3
I20250112 02:52:46.126276 3110 raft_consensus.cc:2238] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:46.126415 3110 raft_consensus.cc:2267] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.126993 3110 ts_tablet_manager.cc:1905] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.128219 746 tablet_service.cc:1467] Processing CreateTablet for tablet 2709ff38dc3e44628bb46d9df9404ee3 (DEFAULT_TABLE table=default.Table_d0f34244c237425c84c19f0c575857e2 [id=5325ba29fd884b31968ee811da6e086a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.128674 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2709ff38dc3e44628bb46d9df9404ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.132753 1181 tablet_bootstrap.cc:492] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:46.132920 1181 ts_tablet_manager.cc:1397] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.001s
I20250112 02:52:46.133602 1181 raft_consensus.cc:357] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:46.133823 1181 raft_consensus.cc:383] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:46.133908 1181 raft_consensus.cc:738] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:46.134148 1181 consensus_queue.cc:260] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:46.134486 3109 ts_tablet_manager.cc:1918] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:46.134583 3109 log.cc:1198] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/bd5b3c712c0f448ea9b14ab93dd731ab
I20250112 02:52:46.134868 3109 ts_tablet_manager.cc:1939] T bd5b3c712c0f448ea9b14ab93dd731ab P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:46.135046 1181 ts_tablet_manager.cc:1428] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:46.135314 1181 tablet_bootstrap.cc:492] T b114b98d1cf4419a8bfeef558d818c01 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:46.136402 1181 tablet_bootstrap.cc:654] T b114b98d1cf4419a8bfeef558d818c01 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:46.139573 3111 ts_tablet_manager.cc:1918] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:46.143776 887 tablet_service.cc:1467] Processing CreateTablet for tablet 12e5e15a95b24794b1f2dd837d548a27 (DEFAULT_TABLE table=default.Table_fed76a4c2ec74241b1c63266f20f6873 [id=849b1709534642e3b466dc6dc5e03374]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.144202 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12e5e15a95b24794b1f2dd837d548a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.144767 1181 tablet_bootstrap.cc:492] T b114b98d1cf4419a8bfeef558d818c01 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:46.144882 1181 ts_tablet_manager.cc:1397] T b114b98d1cf4419a8bfeef558d818c01 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20250112 02:52:46.145380 1181 raft_consensus.cc:357] T b114b98d1cf4419a8bfeef558d818c01 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:46.145565 1181 raft_consensus.cc:383] T b114b98d1cf4419a8bfeef558d818c01 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:46.145619 1181 raft_consensus.cc:738] T b114b98d1cf4419a8bfeef558d818c01 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:46.145802 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 414) took 1340 ms. Trace:
I20250112 02:52:46.146811 3109 rpcz_store.cc:276] 0112 02:52:44.804930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.805532 (+ 602us) service_pool.cc:225] Handling call
0112 02:52:45.992864 (+1187332us) ts_tablet_manager.cc:1145] Deleting tablet bd5b3c712c0f448ea9b14ab93dd731ab
0112 02:52:45.992873 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.048690 (+ 55817us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.134398 (+ 85708us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.145780 (+ 11382us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":107,"tablet-delete.queue_time_us":1187240}
I20250112 02:52:46.148290 1181 consensus_queue.cc:260] T b114b98d1cf4419a8bfeef558d818c01 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:46.149470 1181 ts_tablet_manager.cc:1428] T b114b98d1cf4419a8bfeef558d818c01 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250112 02:52:46.150195 3109 tablet_replica.cc:331] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:46.150379 1181 tablet_bootstrap.cc:492] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:46.150436 3109 raft_consensus.cc:2238] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:46.150650 3109 raft_consensus.cc:2267] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.146732 1173 tablet_bootstrap.cc:492] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:46.150960 1173 ts_tablet_manager.cc:1397] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.099s user 0.003s sys 0.000s
I20250112 02:52:46.151648 1173 raft_consensus.cc:357] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.151882 1173 raft_consensus.cc:383] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:46.151968 1173 raft_consensus.cc:738] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:46.152294 1181 tablet_bootstrap.cc:654] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:46.152197 1173 consensus_queue.cc:260] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.153645 1173 ts_tablet_manager.cc:1428] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 02:52:46.154872 1173 tablet_bootstrap.cc:492] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:46.156500 1173 tablet_bootstrap.cc:654] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:46.157316 3111 log.cc:1198] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/f776e2a6ec704cf9b64483bc7d5b1dff
I20250112 02:52:46.157763 3111 ts_tablet_manager.cc:1939] T f776e2a6ec704cf9b64483bc7d5b1dff P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:46.158627 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 411) took 1040 ms. Trace:
I20250112 02:52:46.158785 3111 rpcz_store.cc:276] 0112 02:52:45.117743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.118636 (+ 893us) service_pool.cc:225] Handling call
0112 02:52:46.050093 (+931457us) ts_tablet_manager.cc:1145] Deleting tablet f776e2a6ec704cf9b64483bc7d5b1dff
0112 02:52:46.050101 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.136472 (+ 86371us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.139484 (+ 3012us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.158605 (+ 19121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":931336}
I20250112 02:52:46.159282 3111 tablet_replica.cc:331] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:46.159386 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet f776e2a6ec704cf9b64483bc7d5b1dff (table default.Table_d0a145ff8f4d4e0484e015fb4d18860c [id=aba940f2fbb849168f3cb1bfd1c36e5e]) successfully deleted
I20250112 02:52:46.159552 3111 raft_consensus.cc:2238] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:46.159888 3111 raft_consensus.cc:2267] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.160799 3111 ts_tablet_manager.cc:1905] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.163300 3109 ts_tablet_manager.cc:1905] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.166712 887 tablet_service.cc:1467] Processing CreateTablet for tablet 61d7703b15024fe7a35805345d5e81ab (DEFAULT_TABLE table=default.Table_bb82d19aab034576ad7a0303219fa410 [id=e1067ea6372e45e0b11b8e5713cd71cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.167182 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61d7703b15024fe7a35805345d5e81ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.170814 1181 tablet_bootstrap.cc:492] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:46.171579 1181 ts_tablet_manager.cc:1397] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.001s
I20250112 02:52:46.172417 1181 raft_consensus.cc:357] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:46.173884 1181 raft_consensus.cc:383] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:46.174142 1181 raft_consensus.cc:738] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:46.174772 1181 consensus_queue.cc:260] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:46.175911 1181 ts_tablet_manager.cc:1428] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250112 02:52:46.176746 1181 tablet_bootstrap.cc:492] T d86417fc1c844cdf85323b94c6dce6d5 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:46.178432 1181 tablet_bootstrap.cc:654] T d86417fc1c844cdf85323b94c6dce6d5 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:46.149972 3110 ts_tablet_manager.cc:1918] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:46.179821 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3a4e76b724452c8b238aae0198e297" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:46.182004 915 raft_consensus.cc:2463] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:46.181990 3177 tablet.cc:1721] T afd6d31155034908ac44a90c3a2ea7d6 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:46.180065 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f16ec377f01484eae10e63490da143f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:46.182348 911 raft_consensus.cc:2463] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:46.181285 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3a4e76b724452c8b238aae0198e297" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:46.182598 914 raft_consensus.cc:3054] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:46.183297 854 leader_election.cc:343] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:46.183390 854 leader_election.cc:304] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:46.183585 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 37d38f5e525440f8a0f7822620933435 with delete_type TABLET_DATA_DELETED (replaced by tablet b1cb48c936844d049a1c832b47986d73) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:46.183622 3000 raft_consensus.cc:2743] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:46.183779 749 tablet_service.cc:1514] Processing DeleteTablet for tablet cd52261c4a414e7d93a528e2eddfa9db with delete_type TABLET_DATA_DELETED (replaced by tablet a7db121c542f423bbffd2e36c459c3f0) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:46.184129 743 tablet_service.cc:1467] Processing CreateTablet for tablet 12e5e15a95b24794b1f2dd837d548a27 (DEFAULT_TABLE table=default.Table_fed76a4c2ec74241b1c63266f20f6873 [id=849b1709534642e3b466dc6dc5e03374]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.184453 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12e5e15a95b24794b1f2dd837d548a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.184867 914 raft_consensus.cc:2463] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:46.185240 3110 log.cc:1198] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/93aba9b4d1dd485cb62c4827c916a1d5
I20250112 02:52:46.185581 3110 ts_tablet_manager.cc:1939] T 93aba9b4d1dd485cb62c4827c916a1d5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:46.186312 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 337) took 1261 ms. Trace:
I20250112 02:52:46.186453 3110 rpcz_store.cc:276] 0112 02:52:44.924788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.925087 (+ 299us) service_pool.cc:225] Handling call
0112 02:52:46.125926 (+1200839us) ts_tablet_manager.cc:1145] Deleting tablet 93aba9b4d1dd485cb62c4827c916a1d5
0112 02:52:46.125937 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.130254 (+ 4317us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.149895 (+ 19641us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.186294 (+ 36399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1200685}
I20250112 02:52:46.186595 3158 consensus_queue.cc:1035] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:46.186947 3110 tablet_replica.cc:331] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:46.187045 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 93aba9b4d1dd485cb62c4827c916a1d5 (table default.Table_d5f4d4c431d24cc0ba82a7dcd012368b [id=3059ab365aff4430b1570224188486fe]) successfully deleted
I20250112 02:52:46.187125 3110 raft_consensus.cc:2238] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:46.187343 3110 raft_consensus.cc:2267] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.187398 989 leader_election.cc:304] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:46.187417 760 tablet_service.cc:1514] Processing DeleteTablet for tablet d2028226498b4b41a95c7c836fe03b4d with delete_type TABLET_DATA_DELETED (replaced by tablet 1b34cef7c8a14dbda7c091dfed46515a) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:46.187697 747 tablet_service.cc:1467] Processing CreateTablet for tablet 61d7703b15024fe7a35805345d5e81ab (DEFAULT_TABLE table=default.Table_bb82d19aab034576ad7a0303219fa410 [id=e1067ea6372e45e0b11b8e5713cd71cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.187813 3158 raft_consensus.cc:2798] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:46.187881 3158 raft_consensus.cc:491] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:46.187934 3158 raft_consensus.cc:3054] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:46.187989 3110 ts_tablet_manager.cc:1905] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.188009 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61d7703b15024fe7a35805345d5e81ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.190282 3158 raft_consensus.cc:513] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.190838 3158 leader_election.cc:290] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:46.191210 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f16ec377f01484eae10e63490da143f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:46.192118 1173 tablet_bootstrap.cc:492] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:46.192214 1173 ts_tablet_manager.cc:1397] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.001s
W20250112 02:52:46.192503 991 leader_election.cc:343] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:46.192740 1173 raft_consensus.cc:357] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.192916 1173 raft_consensus.cc:383] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:46.192956 1173 raft_consensus.cc:738] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:46.193065 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet bd5b3c712c0f448ea9b14ab93dd731ab (table default.Table_9c10574f7c7d435890eb9a24af23bcb9 [id=c0e16b77525146a28b8182137bc81566]) successfully deleted
I20250112 02:52:46.193301 1173 consensus_queue.cc:260] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:46.193784 714 consensus_peers.cc:487] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8 -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7340a7a916c4b2fab32e0ba2d0ce322. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:46.193887 1173 ts_tablet_manager.cc:1428] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:46.194453 1173 tablet_bootstrap.cc:492] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:46.194391 3175 consensus_queue.cc:1035] T fd81852962fb43449e35ef45dc1daaad P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:46.196031 3174 consensus_queue.cc:1035] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:46.196213 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_b9dff6cb7bbd4d0a88de458399b0d2c0" } new_table_name: "default.Table_d3eb6d686d9540ffbdc854d3932aa893" modify_external_catalogs: true
I20250112 02:52:46.196622 1173 tablet_bootstrap.cc:654] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:46.202265 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5b741e4f6abd4695b3ba1d7a667b6046 (table default.Table_50955c16989f4d08bf187803cedb9dad [id=332595f0d3a24adf8b9bbe47b82b9f66]). Expected version 1 got 0
I20250112 02:52:46.202587 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3dab0466476c4a9aa265271c1abd91a4 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:46.202867 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_ce6e0afd36bf48c4937196300248bc26 [id=097d902a92934882bc7c016da518642a]). Expected version 2 got 0
I20250112 02:52:46.203336 451 catalog_manager.cc:5526] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:46.203799 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_ee16ba338f554f4dbd75b332adf897f3 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 2 got 0
I20250112 02:52:46.204035 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:46.204361 451 catalog_manager.cc:5526] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d reported cstate change: term changed from 0 to 1, leader changed from <none> to fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:46.205228 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_a9907ed8bfe14625977ec98b67e9d599 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 2 got 0
I20250112 02:52:46.205431 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 99407ead9cf547e3bbe78c0d0b327e6a (table default.Table_b9dff6cb7bbd4d0a88de458399b0d2c0 [id=d1fa33e075d74f1193fe708f5034fbc1]). Expected version 2 got 0
I20250112 02:52:46.205621 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 5524d75599f04becbe6a94426adb7026 (table default.Table_e8cef38ebaa243ed83bfb41bcea1933f [id=b7739f28885a453d9ef3966511183c17]). Expected version 1 got 0
I20250112 02:52:46.205890 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ca2165362ee541adac724b63f39b1489 (table default.Table_41b7056b6e9240928d0af52aec2e9338 [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 2 got 0
I20250112 02:52:46.206023 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 56ddc025a8314f6f84d229191eb56692 (table default.Table_d22db1cc1d354af495d9888de1bfc22a [id=2ddd5dd296b94bc8b96a97b66e775836]). Expected version 1 got 0
I20250112 02:52:46.206156 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:46.206665 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:46.206868 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_1506fd71c54a4312b1c439a2effa4c69 [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 3 got 0
I20250112 02:52:46.207355 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 47a0d939fdb94590a86389f1f4693e9a (table default.Table_ac4a10685c414bef86c147c336335ff7 [id=78ee5f3a07714849a10192e80c425264]). Expected version 2 got 1
I20250112 02:52:46.207645 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_37bd8ec69e20493b8d2685e143a51327 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 2 got 0
I20250112 02:52:46.207978 3109 ts_tablet_manager.cc:1918] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:46.208020 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 0882ea69a90b4ce3abb4f2da6475972f (table default.Table_476e5893d48e49a9a3fa17be765213d9 [id=45d7bbac986d4db9a3e0b6f1a88ffa1d]). Expected version 1 got 0
I20250112 02:52:46.208088 3109 log.cc:1198] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/f776e2a6ec704cf9b64483bc7d5b1dff
I20250112 02:52:46.208415 3109 ts_tablet_manager.cc:1939] T f776e2a6ec704cf9b64483bc7d5b1dff P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:46.208791 2704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "436220c23fa948efb5b6ca15875d8b0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436220c23fa948efb5b6ca15875d8b0e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42341 } } peers { permanent_uuid: "f4a49a5756b143fcb767c3f41463f8d9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38771 } } peers { permanent_uuid: "dd25f03f25df49178a5c53c636ecf304" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39987 } } }
I20250112 02:52:46.209053 2704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e [sys.catalog]: This master's current role is: LEADER
I20250112 02:52:46.209270 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 417) took 1404 ms. Trace:
I20250112 02:52:46.209348 3109 rpcz_store.cc:276] 0112 02:52:44.805168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.805889 (+ 721us) service_pool.cc:225] Handling call
0112 02:52:46.150146 (+1344257us) ts_tablet_manager.cc:1145] Deleting tablet f776e2a6ec704cf9b64483bc7d5b1dff
0112 02:52:46.150159 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.166554 (+ 16395us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.207878 (+ 41324us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.209246 (+ 1368us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1344128}
I20250112 02:52:46.209833 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 418) took 1404 ms. Trace:
I20250112 02:52:46.209836 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet f776e2a6ec704cf9b64483bc7d5b1dff (table default.Table_d0a145ff8f4d4e0484e015fb4d18860c [id=aba940f2fbb849168f3cb1bfd1c36e5e]) successfully deleted
I20250112 02:52:46.209923 3109 rpcz_store.cc:276] 0112 02:52:44.805251 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.806036 (+ 785us) service_pool.cc:225] Handling call
0112 02:52:46.209680 (+1403644us) ts_tablet_manager.cc:1145] Deleting tablet a8e7882e07784bdda18dc1ffd20963c8
0112 02:52:46.209706 (+ 26us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.209815 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1403537}
I20250112 02:52:46.210228 3109 tablet_replica.cc:331] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
W20250112 02:52:46.210395 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet a8e7882e07784bdda18dc1ffd20963c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8e7882e07784bdda18dc1ffd20963c8 already in progress: creating tablet
I20250112 02:52:46.210413 3109 raft_consensus.cc:2238] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:46.210708 3109 raft_consensus.cc:2267] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.213120 3110 ts_tablet_manager.cc:1918] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:46.213248 3110 log.cc:1198] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/f3771166b77e4f90aaa116ab1a9dc299
I20250112 02:52:46.213570 3110 ts_tablet_manager.cc:1939] T f3771166b77e4f90aaa116ab1a9dc299 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:46.214417 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 331) took 1345 ms. Trace:
I20250112 02:52:46.214514 3110 rpcz_store.cc:276] 0112 02:52:44.868837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.868884 (+ 47us) service_pool.cc:225] Handling call
0112 02:52:46.186911 (+1318027us) ts_tablet_manager.cc:1145] Deleting tablet f3771166b77e4f90aaa116ab1a9dc299
0112 02:52:46.186919 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.190541 (+ 3622us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.213017 (+ 22476us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.214396 (+ 1379us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1257781}
I20250112 02:52:46.215009 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet f3771166b77e4f90aaa116ab1a9dc299 (table default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6]) successfully deleted
I20250112 02:52:46.217875 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 61d7703b15024fe7a35805345d5e81ab (DEFAULT_TABLE table=default.Table_bb82d19aab034576ad7a0303219fa410 [id=e1067ea6372e45e0b11b8e5713cd71cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.218276 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61d7703b15024fe7a35805345d5e81ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.218708 747 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea00cab2f6f44b58d82c02a07513416 (DEFAULT_TABLE table=default.Table_87c7c83e6cef4175a123fa9924acd6d7 [id=7301a1efa6e8465eaf5d21fa43d25c2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.219000 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ea00cab2f6f44b58d82c02a07513416. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.223794 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea00cab2f6f44b58d82c02a07513416 (DEFAULT_TABLE table=default.Table_87c7c83e6cef4175a123fa9924acd6d7 [id=7301a1efa6e8465eaf5d21fa43d25c2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.224205 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ea00cab2f6f44b58d82c02a07513416. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.227953 895 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea00cab2f6f44b58d82c02a07513416 (DEFAULT_TABLE table=default.Table_87c7c83e6cef4175a123fa9924acd6d7 [id=7301a1efa6e8465eaf5d21fa43d25c2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.229420 1177 tablet_bootstrap.cc:492] T 125b12a2f9164653be79d3e6d8313d36 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:46.229561 1177 ts_tablet_manager.cc:1397] T 125b12a2f9164653be79d3e6d8313d36 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.191s user 0.013s sys 0.030s
I20250112 02:52:46.230229 1177 raft_consensus.cc:357] T 125b12a2f9164653be79d3e6d8313d36 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.230410 1177 raft_consensus.cc:383] T 125b12a2f9164653be79d3e6d8313d36 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:46.230471 1177 raft_consensus.cc:738] T 125b12a2f9164653be79d3e6d8313d36 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:46.230644 1177 consensus_queue.cc:260] T 125b12a2f9164653be79d3e6d8313d36 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.231063 1177 ts_tablet_manager.cc:1428] T 125b12a2f9164653be79d3e6d8313d36 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:46.231288 1177 tablet_bootstrap.cc:492] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:46.235447 3111 ts_tablet_manager.cc:1918] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:46.235796 3111 log.cc:1198] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/62f4bd2df0ce45dabc55373f1154d3ac
I20250112 02:52:46.236513 1173 tablet_bootstrap.cc:492] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:46.237066 1173 ts_tablet_manager.cc:1397] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.001s
I20250112 02:52:46.237813 1173 raft_consensus.cc:357] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.238020 1173 raft_consensus.cc:383] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:46.238121 1173 raft_consensus.cc:738] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:46.243610 1173 consensus_queue.cc:260] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.244304 1173 ts_tablet_manager.cc:1428] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250112 02:52:46.244328 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f16ec377f01484eae10e63490da143f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:46.244484 916 raft_consensus.cc:3054] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:46.244575 1173 tablet_bootstrap.cc:492] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:46.244941 3111 ts_tablet_manager.cc:1939] T 62f4bd2df0ce45dabc55373f1154d3ac P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:46.245174 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 412) took 1127 ms. Trace:
I20250112 02:52:46.245227 3111 rpcz_store.cc:276] 0112 02:52:45.117839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.118787 (+ 948us) service_pool.cc:225] Handling call
0112 02:52:46.159233 (+1040446us) ts_tablet_manager.cc:1145] Deleting tablet 62f4bd2df0ce45dabc55373f1154d3ac
0112 02:52:46.159245 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.205743 (+ 46498us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.235374 (+ 29631us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.245164 (+ 9790us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1040303}
I20250112 02:52:46.245561 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 62f4bd2df0ce45dabc55373f1154d3ac (table default.Table_7921b99e24f84cceb58128f18925ad49 [id=0525f87eccbb4a1d8726d18a00f9a4e4]) successfully deleted
I20250112 02:52:46.246116 3109 ts_tablet_manager.cc:1905] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.246760 916 raft_consensus.cc:2463] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:46.248842 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ea00cab2f6f44b58d82c02a07513416. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.252204 3111 tablet_replica.cc:331] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:46.252435 3111 raft_consensus.cc:2238] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:46.252760 3111 raft_consensus.cc:2267] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.253353 3111 ts_tablet_manager.cc:1905] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.253657 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 339) took 1254 ms. Trace:
I20250112 02:52:46.253803 3110 rpcz_store.cc:276] 0112 02:52:44.998915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.998958 (+ 43us) service_pool.cc:225] Handling call
0112 02:52:46.253562 (+1254604us) ts_tablet_manager.cc:1145] Deleting tablet 1a7717cdcb8a4f47b11ba7af54b57e3c
0112 02:52:46.253571 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.253643 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1254478}
I20250112 02:52:46.254269 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 347) took 1118 ms. Trace:
I20250112 02:52:46.254397 3110 rpcz_store.cc:276] 0112 02:52:45.135313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.135716 (+ 403us) service_pool.cc:225] Handling call
0112 02:52:46.254161 (+1118445us) ts_tablet_manager.cc:1145] Deleting tablet 3d662e0ce13f447595e5f92b94de3018
0112 02:52:46.254168 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.254256 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1118278}
W20250112 02:52:46.254907 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 1a7717cdcb8a4f47b11ba7af54b57e3c because the tablet was not found. No further retry: Not found: Tablet not found: 1a7717cdcb8a4f47b11ba7af54b57e3c
W20250112 02:52:46.255146 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 3d662e0ce13f447595e5f92b94de3018 because the tablet was not found. No further retry: Not found: Tablet not found: 3d662e0ce13f447595e5f92b94de3018
I20250112 02:52:46.255416 1177 tablet_bootstrap.cc:654] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:46.255388 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 348) took 1119 ms. Trace:
I20250112 02:52:46.258219 3110 rpcz_store.cc:276] 0112 02:52:45.135418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.135911 (+ 493us) service_pool.cc:225] Handling call
0112 02:52:46.255019 (+1119108us) ts_tablet_manager.cc:1145] Deleting tablet 2361bc6bd8164c2daa37be6b0845f4bd
0112 02:52:46.255117 (+ 98us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.255272 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1118971}
I20250112 02:52:46.258662 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 350) took 1093 ms. Trace:
I20250112 02:52:46.258772 3110 rpcz_store.cc:276] 0112 02:52:45.164834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.164980 (+ 146us) service_pool.cc:225] Handling call
0112 02:52:46.258574 (+1093594us) ts_tablet_manager.cc:1145] Deleting tablet 4cfe1a1595014c27bbd350cec169d71a
0112 02:52:46.258582 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.258649 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1093435}
I20250112 02:52:46.259249 3110 tablet_replica.cc:331] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:46.259521 3110 raft_consensus.cc:2238] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:46.259708 3110 raft_consensus.cc:2267] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.260197 3110 ts_tablet_manager.cc:1905] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:46.260975 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 2361bc6bd8164c2daa37be6b0845f4bd because the tablet was not found. No further retry: Not found: Tablet not found: 2361bc6bd8164c2daa37be6b0845f4bd
W20250112 02:52:46.261252 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 4cfe1a1595014c27bbd350cec169d71a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cfe1a1595014c27bbd350cec169d71a already in progress: creating tablet
I20250112 02:52:46.261665 989 leader_election.cc:304] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:46.262065 2745 raft_consensus.cc:2798] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:46.262223 2745 raft_consensus.cc:695] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:46.262688 2745 consensus_queue.cc:237] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.265581 1181 tablet_bootstrap.cc:492] T d86417fc1c844cdf85323b94c6dce6d5 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:46.265698 1181 ts_tablet_manager.cc:1397] T d86417fc1c844cdf85323b94c6dce6d5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.002s
I20250112 02:52:46.208889 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b455a37448a044679bd2c0646de3d804 (table default.Table_9ecfcdf0c66d4e3184fae185f8fe6c9e [id=55eb92db6e924911b44fb44221d453e3]). Expected version 1 got 0
I20250112 02:52:46.266264 1181 raft_consensus.cc:357] T d86417fc1c844cdf85323b94c6dce6d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:46.266543 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 2c03b5e612d64314b210c11686fdfa2a (table default.Table_ebbae2e0c50f44fd9e379196c2be90af [id=5c4769bfcefd4fe0a0f3c2e315d50c32]). Expected version 7 got 6
I20250112 02:52:46.260689 3106 consensus_queue.cc:1035] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:46.262637 3109 ts_tablet_manager.cc:1918] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:46.269816 3109 log.cc:1198] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/d63d411f906b4d4eb68343bdcc493e05
I20250112 02:52:46.270187 3109 ts_tablet_manager.cc:1939] T d63d411f906b4d4eb68343bdcc493e05 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:46.270922 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 412) took 1466 ms. Trace:
I20250112 02:52:46.264963 3138 consensus_queue.cc:1035] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:46.272368 3109 rpcz_store.cc:276] 0112 02:52:44.804725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.806243 (+ 1518us) service_pool.cc:225] Handling call
0112 02:52:46.210180 (+1403937us) ts_tablet_manager.cc:1145] Deleting tablet d63d411f906b4d4eb68343bdcc493e05
0112 02:52:46.210191 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.257128 (+ 46937us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.262536 (+ 5408us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.270905 (+ 8369us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1403821}
I20250112 02:52:46.272635 3109 tablet_replica.cc:331] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:46.272790 3109 raft_consensus.cc:2238] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:46.272887 3109 raft_consensus.cc:2267] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.273341 3109 ts_tablet_manager.cc:1905] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.276176 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet d63d411f906b4d4eb68343bdcc493e05 (table default.Table_e80769df10444dd5ae4bc23a8c6cc09e [id=d91a9f9e1f76482e99d4f10c0568f2e3]) successfully deleted
I20250112 02:52:46.277524 3109 ts_tablet_manager.cc:1918] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:46.277662 3109 log.cc:1198] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/8429a3d8636d4e2ca4c1191f85b9266b
I20250112 02:52:46.277987 3109 ts_tablet_manager.cc:1939] T 8429a3d8636d4e2ca4c1191f85b9266b P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:46.278695 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 419) took 1473 ms. Trace:
I20250112 02:52:46.271688 3110 ts_tablet_manager.cc:1918] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:46.278859 3110 log.cc:1198] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/ea33053c62dc4707b759d5c1756ab4bd
I20250112 02:52:46.278836 3109 rpcz_store.cc:276] 0112 02:52:44.805324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.809755 (+ 4431us) service_pool.cc:225] Handling call
0112 02:52:46.272601 (+1462846us) ts_tablet_manager.cc:1145] Deleting tablet 8429a3d8636d4e2ca4c1191f85b9266b
0112 02:52:46.272608 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.275119 (+ 2511us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.277463 (+ 2344us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.278677 (+ 1214us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1462697}
I20250112 02:52:46.279126 3109 tablet_replica.cc:331] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:46.279137 3110 ts_tablet_manager.cc:1939] T ea33053c62dc4707b759d5c1756ab4bd P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:46.279304 3109 raft_consensus.cc:2238] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:46.279484 3109 pending_rounds.cc:62] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:46.279521 3109 pending_rounds.cc:69] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319898055630848 op_type: NO_OP noop_request { }
I20250112 02:52:46.279600 3109 raft_consensus.cc:2883] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:46.279662 3109 raft_consensus.cc:2267] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.279755 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 351) took 1098 ms. Trace:
I20250112 02:52:46.279817 3110 rpcz_store.cc:276] 0112 02:52:45.181379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.181800 (+ 421us) service_pool.cc:225] Handling call
0112 02:52:46.259215 (+1077415us) ts_tablet_manager.cc:1145] Deleting tablet ea33053c62dc4707b759d5c1756ab4bd
0112 02:52:46.259222 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.267695 (+ 8473us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.271623 (+ 3928us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.279740 (+ 8117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1077269}
I20250112 02:52:46.280086 3110 tablet_replica.cc:331] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:46.280236 3110 raft_consensus.cc:2238] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:46.280313 3109 ts_tablet_manager.cc:1905] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.280341 3110 raft_consensus.cc:2267] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.280735 3110 ts_tablet_manager.cc:1905] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.282249 1173 tablet_bootstrap.cc:654] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:46.284152 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 8429a3d8636d4e2ca4c1191f85b9266b (table default.Table_328664e3c01f483ca8e5ff50a497f68e [id=1f5c5f9f5e664250898a3935787b6b51]) successfully deleted
I20250112 02:52:46.288883 3000 raft_consensus.cc:491] T f29fb4cc96f64cbeabad085f1007716f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:46.289003 3000 raft_consensus.cc:513] T f29fb4cc96f64cbeabad085f1007716f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:46.289573 3000 leader_election.cc:290] T f29fb4cc96f64cbeabad085f1007716f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:46.289669 3109 ts_tablet_manager.cc:1918] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:46.289762 3109 log.cc:1198] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/ff1b3ee229c445b19405cf4a19cf8003
I20250112 02:52:46.290064 3109 ts_tablet_manager.cc:1939] T ff1b3ee229c445b19405cf4a19cf8003 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:46.290228 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f29fb4cc96f64cbeabad085f1007716f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:46.290704 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 421) took 1473 ms. Trace:
W20250112 02:52:46.290704 855 leader_election.cc:343] T f29fb4cc96f64cbeabad085f1007716f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:46.290815 3109 rpcz_store.cc:276] 0112 02:52:44.817334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.817393 (+ 59us) service_pool.cc:225] Handling call
0112 02:52:46.279075 (+1461682us) ts_tablet_manager.cc:1145] Deleting tablet ff1b3ee229c445b19405cf4a19cf8003
0112 02:52:46.279085 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.285388 (+ 6303us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.289599 (+ 4211us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.290690 (+ 1091us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1461538}
I20250112 02:52:46.291253 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 422) took 1473 ms. Trace:
I20250112 02:52:46.291328 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet ff1b3ee229c445b19405cf4a19cf8003 (table default.Table_6b8cb70a5e5b4519af5d247f6b5bb8b0 [id=a28f5cfbcfe64768b40477ffc2053e27]) successfully deleted
I20250112 02:52:46.291369 3109 rpcz_store.cc:276] 0112 02:52:44.817535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.817597 (+ 62us) service_pool.cc:225] Handling call
0112 02:52:46.291128 (+1473531us) ts_tablet_manager.cc:1145] Deleting tablet 23e7a2e88c374e949ced92c557225cbf
0112 02:52:46.291137 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.291239 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1473403}
I20250112 02:52:46.291632 3109 tablet_replica.cc:331] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
W20250112 02:52:46.291760 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 23e7a2e88c374e949ced92c557225cbf because the tablet was not found. No further retry: Not found: Tablet not found: 23e7a2e88c374e949ced92c557225cbf
I20250112 02:52:46.291873 3109 raft_consensus.cc:2238] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:46.291908 3110 ts_tablet_manager.cc:1918] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:46.291987 3110 log.cc:1198] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/a4328137407843e1879493ef0c56ffc8
I20250112 02:52:46.292070 3109 raft_consensus.cc:2267] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.292320 3110 ts_tablet_manager.cc:1939] T a4328137407843e1879493ef0c56ffc8 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:46.292589 3109 ts_tablet_manager.cc:1905] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.293058 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 352) took 1111 ms. Trace:
I20250112 02:52:46.293179 3110 rpcz_store.cc:276] 0112 02:52:45.181490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.181978 (+ 488us) service_pool.cc:225] Handling call
0112 02:52:46.280050 (+1098072us) ts_tablet_manager.cc:1145] Deleting tablet a4328137407843e1879493ef0c56ffc8
0112 02:52:46.280058 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.283212 (+ 3154us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.291836 (+ 8624us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.293039 (+ 1203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1097952}
I20250112 02:52:46.293639 3110 tablet_replica.cc:331] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:46.293860 3110 raft_consensus.cc:2238] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:46.296042 3110 raft_consensus.cc:2267] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.297969 3110 ts_tablet_manager.cc:1905] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.300420 3158 raft_consensus.cc:491] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:46.300545 3158 raft_consensus.cc:513] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.301134 3158 leader_election.cc:290] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:46.301855 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95f98ae7bdf42518bbff55e257fea88" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:46.302356 991 leader_election.cc:343] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:46.302922 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95f98ae7bdf42518bbff55e257fea88" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:46.303242 917 raft_consensus.cc:2463] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:46.303675 1181 raft_consensus.cc:383] T d86417fc1c844cdf85323b94c6dce6d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:46.303975 1181 raft_consensus.cc:738] T d86417fc1c844cdf85323b94c6dce6d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:46.303997 989 leader_election.cc:304] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:46.304430 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f29fb4cc96f64cbeabad085f1007716f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:46.304405 1181 consensus_queue.cc:260] T d86417fc1c844cdf85323b94c6dce6d5 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:46.304874 854 leader_election.cc:343] T f29fb4cc96f64cbeabad085f1007716f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:46.304914 3158 raft_consensus.cc:2798] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:46.304987 3158 raft_consensus.cc:491] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:46.305047 3158 raft_consensus.cc:3054] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:46.305040 854 leader_election.cc:304] T f29fb4cc96f64cbeabad085f1007716f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:46.305874 3000 raft_consensus.cc:2743] T f29fb4cc96f64cbeabad085f1007716f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:46.307262 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:46.308028 1181 ts_tablet_manager.cc:1428] T d86417fc1c844cdf85323b94c6dce6d5 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.042s user 0.001s sys 0.001s
I20250112 02:52:46.307930 3158 raft_consensus.cc:513] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.308885 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95f98ae7bdf42518bbff55e257fea88" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:46.309157 2978 consensus_queue.cc:1035] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:52:46.309291 991 leader_election.cc:343] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:46.309927 3158 leader_election.cc:290] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:46.310514 3111 ts_tablet_manager.cc:1918] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:46.310688 3111 log.cc:1198] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/54215c55966a46e1ab96e352c4e3764e
I20250112 02:52:46.310707 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:46.311014 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet abfaad67d5e34c0a8338dccc0ef05466 (table default.Table_16821effaf954ac28a375fb933aef7d3 [id=2c0e94e8c5774da1b8130c7651d94500]). Expected version 2 got 1
I20250112 02:52:46.311187 3111 ts_tablet_manager.cc:1939] T 54215c55966a46e1ab96e352c4e3764e P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:46.311324 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3f5b418c17084420b867498fda79fb0a (table default.Table_b0b0d76c3b4a428f9ffd93ebb6e96770 [id=ad0be37e6b6f4b9597dcb4cbd50d5b12]). Expected version 4 got 3
I20250112 02:52:46.311578 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:46.311595 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 413) took 1193 ms. Trace:
I20250112 02:52:46.311933 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet e3ee5d8a17874d7ba54de51568e7d2e6 (table default.Table_630d49265f684b0dbef983cb09ea0a14 [id=2c077da077834ae4bcc84efdd9e89120]). Expected version 2 got 1
I20250112 02:52:46.312516 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet afd6d31155034908ac44a90c3a2ea7d6 (table default.Table_ebadf77c864c4883a04b9f62f15d7f49 [id=5cec72cf833347f3bfbca061f8feca41]). Expected version 1 got 0
I20250112 02:52:46.312515 3111 rpcz_store.cc:276] 0112 02:52:45.117936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.118955 (+ 1019us) service_pool.cc:225] Handling call
0112 02:52:46.252155 (+1133200us) ts_tablet_manager.cc:1145] Deleting tablet 54215c55966a46e1ab96e352c4e3764e
0112 02:52:46.252173 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.284937 (+ 32764us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.310419 (+ 25482us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.311581 (+ 1162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1128538}
I20250112 02:52:46.313098 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 54215c55966a46e1ab96e352c4e3764e (table default.Table_6e77252e817f43399c9b3f5b221795ad [id=1fa1474ab8804667b22e5f30c48c1630]) successfully deleted
I20250112 02:52:46.313473 3111 tablet_replica.cc:331] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:46.313836 3111 raft_consensus.cc:2238] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:46.314150 3111 pending_rounds.cc:62] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6: Trying to abort 1 pending ops.
I20250112 02:52:46.314258 3111 pending_rounds.cc:69] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319891534884864 op_type: NO_OP noop_request { }
I20250112 02:52:46.314386 3111 raft_consensus.cc:2883] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:46.314499 3111 raft_consensus.cc:2267] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.315026 3111 ts_tablet_manager.cc:1905] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.321391 3109 ts_tablet_manager.cc:1918] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:46.333855 3109 log.cc:1198] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/3b9280db29e54d41af1bed2fa5aa5535
I20250112 02:52:46.334365 3109 ts_tablet_manager.cc:1939] T 3b9280db29e54d41af1bed2fa5aa5535 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:46.334563 3111 ts_tablet_manager.cc:1918] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:46.334748 3111 log.cc:1198] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/a3ffdc237f8640f0b378809f93fc6292
I20250112 02:52:46.335119 3111 ts_tablet_manager.cc:1939] T a3ffdc237f8640f0b378809f93fc6292 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:46.335490 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 423) took 1517 ms. Trace:
I20250112 02:52:46.337069 3109 rpcz_store.cc:276] 0112 02:52:44.818053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.818346 (+ 293us) service_pool.cc:225] Handling call
0112 02:52:46.291594 (+1473248us) ts_tablet_manager.cc:1145] Deleting tablet 3b9280db29e54d41af1bed2fa5aa5535
0112 02:52:46.291602 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.318931 (+ 27329us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.321330 (+ 2399us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.335446 (+ 14116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1473075}
I20250112 02:52:46.337873 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 3b9280db29e54d41af1bed2fa5aa5535 (table default.Table_6671f628f89b428a83eca77e9ca26516 [id=e0577e1a40064437bb7eb4c66fbd914e]) successfully deleted
I20250112 02:52:46.338008 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 424) took 1519 ms. Trace:
I20250112 02:52:46.335433 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 416) took 1204 ms. Trace:
I20250112 02:52:46.338076 3109 rpcz_store.cc:276] 0112 02:52:44.818301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.818361 (+ 60us) service_pool.cc:225] Handling call
0112 02:52:46.337879 (+1519518us) ts_tablet_manager.cc:1145] Deleting tablet 83c04f1a2cb04950a9f6170e252c19b4
0112 02:52:46.337891 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.337994 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1519285}
I20250112 02:52:46.338207 3111 rpcz_store.cc:276] 0112 02:52:45.130643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.131153 (+ 510us) service_pool.cc:225] Handling call
0112 02:52:46.313423 (+1182270us) ts_tablet_manager.cc:1145] Deleting tablet a3ffdc237f8640f0b378809f93fc6292
0112 02:52:46.313435 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.317958 (+ 4523us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.334472 (+ 16514us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.335418 (+ 946us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1182093}
I20250112 02:52:46.338732 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet a3ffdc237f8640f0b378809f93fc6292 (table default.Table_fc264b6a06584c7ea757f5d18e05e2bc [id=b560a3488e3d4db58abc7d81d6094d86]) successfully deleted
W20250112 02:52:46.339116 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 83c04f1a2cb04950a9f6170e252c19b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83c04f1a2cb04950a9f6170e252c19b4 already in progress: creating tablet
I20250112 02:52:46.339397 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 425) took 1520 ms. Trace:
I20250112 02:52:46.339450 3111 tablet_replica.cc:331] T d63d411f906b4d4eb68343bdcc493e05 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:46.339785 3111 raft_consensus.cc:2238] T d63d411f906b4d4eb68343bdcc493e05 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:46.339769 3109 rpcz_store.cc:276] 0112 02:52:44.818815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.818860 (+ 45us) service_pool.cc:225] Handling call
0112 02:52:46.339298 (+1520438us) ts_tablet_manager.cc:1145] Deleting tablet 781a9943f705477cb6eae16df4468d2f
0112 02:52:46.339307 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.339384 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1520307}
I20250112 02:52:46.339996 3111 raft_consensus.cc:2267] T d63d411f906b4d4eb68343bdcc493e05 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.340440 3109 tablet_replica.cc:331] T 82b351c563d749df806ab5d0331a4ef2 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
W20250112 02:52:46.340551 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 781a9943f705477cb6eae16df4468d2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 781a9943f705477cb6eae16df4468d2f already in progress: creating tablet
I20250112 02:52:46.340893 3109 raft_consensus.cc:2238] T 82b351c563d749df806ab5d0331a4ef2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:46.341873 3111 ts_tablet_manager.cc:1905] T d63d411f906b4d4eb68343bdcc493e05 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.343312 3109 raft_consensus.cc:2267] T 82b351c563d749df806ab5d0331a4ef2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.344061 3109 ts_tablet_manager.cc:1905] T 82b351c563d749df806ab5d0331a4ef2 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.353869 3109 ts_tablet_manager.cc:1918] T 82b351c563d749df806ab5d0331a4ef2 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:46.355062 3000 raft_consensus.cc:491] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:46.355222 3000 raft_consensus.cc:513] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.355998 3000 leader_election.cc:290] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:46.357210 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95f98ae7bdf42518bbff55e257fea88" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:46.357412 917 raft_consensus.cc:3054] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:46.359869 917 raft_consensus.cc:2463] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:46.360518 1181 tablet_bootstrap.cc:492] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:46.361002 3111 ts_tablet_manager.cc:1918] T d63d411f906b4d4eb68343bdcc493e05 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:46.361179 3111 log.cc:1198] T d63d411f906b4d4eb68343bdcc493e05 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/d63d411f906b4d4eb68343bdcc493e05
I20250112 02:52:46.362298 1181 tablet_bootstrap.cc:654] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:46.363871 3111 ts_tablet_manager.cc:1939] T d63d411f906b4d4eb68343bdcc493e05 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:46.364192 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 415) took 1233 ms. Trace:
I20250112 02:52:46.364308 3111 rpcz_store.cc:276] 0112 02:52:45.130493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.131365 (+ 872us) service_pool.cc:225] Handling call
0112 02:52:46.339403 (+1208038us) ts_tablet_manager.cc:1145] Deleting tablet d63d411f906b4d4eb68343bdcc493e05
0112 02:52:46.339414 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.348091 (+ 8677us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.360907 (+ 12816us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.364180 (+ 3273us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1207861}
I20250112 02:52:46.364754 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet d63d411f906b4d4eb68343bdcc493e05 (table default.Table_e80769df10444dd5ae4bc23a8c6cc09e [id=d91a9f9e1f76482e99d4f10c0568f2e3]) successfully deleted
I20250112 02:52:46.365181 3111 tablet_replica.cc:331] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:46.365372 3111 raft_consensus.cc:2238] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:46.365622 3111 pending_rounds.cc:62] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6: Trying to abort 1 pending ops.
I20250112 02:52:46.365698 3111 pending_rounds.cc:69] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319896667328512 op_type: NO_OP noop_request { }
I20250112 02:52:46.365833 3111 raft_consensus.cc:2883] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:46.365928 3111 raft_consensus.cc:2267] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.366756 3111 ts_tablet_manager.cc:1905] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.313099 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 05000c6a48e0450d8e6685ca864f67f0 (table default.Table_dcdbe8e16ced4132a8b540781fd01d2e [id=1ec97dc2b50a4a3dbbc449559938cead]). Expected version 1 got 0
I20250112 02:52:46.370306 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 17c54c5dc86446cf85974777c8145ef6 (table default.Table_8ab566cf9c8f45eba4dc9bb85264c823 [id=a622f7a973344a388b3275adbcce6234]). Expected version 1 got 0
I20250112 02:52:46.370618 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 37862023095644bab2403576e8cb8c58 (table default.Table_aef01bf228ad41138e2ffe97ce0e906a [id=01ca9afae4dc47499802f76b7ee93bdf]). Expected version 1 got 0
I20250112 02:52:46.371595 451 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:46.371964 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 5804fc4130f84c69832c1022b7bd7fa5
I20250112 02:52:46.372097 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 3499374810fb412e938325434c35618d
I20250112 02:52:46.372750 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_a27a8c5a296a419fba353c1477c47d18" } modify_external_catalogs: true
I20250112 02:52:46.372809 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_37bd8ec69e20493b8d2685e143a51327" } new_table_name: "default.Table_9f558b53671c40849c3fe1c50dba42d8" modify_external_catalogs: true
I20250112 02:52:46.372938 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_a27a8c5a296a419fba353c1477c47d18" } modify_external_catalogs: true
I20250112 02:52:46.373975 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet b3c5a861a90840d0a33b780ec75bffd2
W20250112 02:52:46.374025 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8fe57d374eb44c3fb1532484e214b654 consensus state has no leader
I20250112 02:52:46.374481 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_872ea90a0f274baea3d3d36c306f2e35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:46.374622 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_518ec5f7946342419e87f093585388fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.374969 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_872ea90a0f274baea3d3d36c306f2e35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:46.375166 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_518ec5f7946342419e87f093585388fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.356606 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d47f6d663934b25a0200ee9fa81165d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:46.375847 3109 log.cc:1198] T 82b351c563d749df806ab5d0331a4ef2 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/82b351c563d749df806ab5d0331a4ef2
I20250112 02:52:46.376235 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 61d7703b15024fe7a35805345d5e81ab
I20250112 02:52:46.376245 3109 ts_tablet_manager.cc:1939] T 82b351c563d749df806ab5d0331a4ef2 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:46.376950 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 426) took 1557 ms. Trace:
I20250112 02:52:46.377018 3109 rpcz_store.cc:276] 0112 02:52:44.819386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.819438 (+ 52us) service_pool.cc:225] Handling call
0112 02:52:46.340390 (+1520952us) ts_tablet_manager.cc:1145] Deleting tablet 82b351c563d749df806ab5d0331a4ef2
0112 02:52:46.340401 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.351186 (+ 10785us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.353792 (+ 2606us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.376931 (+ 23139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1520783}
I20250112 02:52:46.377002 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_1378877b3ffb4d5a95f50b55b1bf1297"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.377470 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1378877b3ffb4d5a95f50b55b1bf1297 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.377341 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_29f995195bb64f709d7b2eee97f35454"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.377925 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29f995195bb64f709d7b2eee97f35454 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.378954 3111 ts_tablet_manager.cc:1918] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:46.379052 3111 log.cc:1198] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/13578542e9e741e8817e0ee63b10f1a1
W20250112 02:52:46.372110 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99407ead9cf547e3bbe78c0d0b327e6a consensus state has no leader
I20250112 02:52:46.379626 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_6186d166c9594cfea3e160905f3737f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.379976 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6186d166c9594cfea3e160905f3737f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.380014 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 3499374810fb412e938325434c35618d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:46.380074 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 82b351c563d749df806ab5d0331a4ef2 (table default.Table_4603068c01544e2e965dca5177e36b40 [id=8502f21e9b61436093db1241cfbb5ec9]) successfully deleted
I20250112 02:52:46.380234 743 tablet_service.cc:1514] Processing DeleteTablet for tablet b3c5a861a90840d0a33b780ec75bffd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:46.380383 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 61d7703b15024fe7a35805345d5e81ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:46.380481 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 5804fc4130f84c69832c1022b7bd7fa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:46.381594 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 427) took 1561 ms. Trace:
I20250112 02:52:46.381873 3109 rpcz_store.cc:276] 0112 02:52:44.819874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.820172 (+ 298us) service_pool.cc:225] Handling call
0112 02:52:46.381070 (+1560898us) ts_tablet_manager.cc:1145] Deleting tablet 681a930541a84f0197ecafe670278e70
0112 02:52:46.381078 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.381580 (+ 502us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1560759}
I20250112 02:52:46.382531 3111 ts_tablet_manager.cc:1939] T 13578542e9e741e8817e0ee63b10f1a1 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:46.382634 747 tablet_service.cc:1514] Processing DeleteTablet for tablet f776e2a6ec704cf9b64483bc7d5b1dff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:29 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:46.383734 3109 tablet_replica.cc:331] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
W20250112 02:52:46.383698 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 681a930541a84f0197ecafe670278e70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 681a930541a84f0197ecafe670278e70 already in progress: creating tablet
I20250112 02:52:46.382675 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 5804fc4130f84c69832c1022b7bd7fa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.382807 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 3499374810fb412e938325434c35618d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.382903 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet b3c5a861a90840d0a33b780ec75bffd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.386359 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 417) took 1255 ms. Trace:
I20250112 02:52:46.387826 3109 raft_consensus.cc:2238] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:46.388234 3109 pending_rounds.cc:62] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:46.388324 3109 pending_rounds.cc:69] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319896667328512 op_type: NO_OP noop_request { }
I20250112 02:52:46.388456 3109 raft_consensus.cc:2883] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:46.388664 3111 rpcz_store.cc:276] 0112 02:52:45.130773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.131568 (+ 795us) service_pool.cc:225] Handling call
0112 02:52:46.365140 (+1233572us) ts_tablet_manager.cc:1145] Deleting tablet 13578542e9e741e8817e0ee63b10f1a1
0112 02:52:46.365149 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.369484 (+ 4335us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.378858 (+ 9374us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.386343 (+ 7485us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1233431}
I20250112 02:52:46.388952 3109 raft_consensus.cc:2267] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.389496 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 418) took 1258 ms. Trace:
I20250112 02:52:46.390285 3109 ts_tablet_manager.cc:1905] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.390614 3111 rpcz_store.cc:276] 0112 02:52:45.130871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.131734 (+ 863us) service_pool.cc:225] Handling call
0112 02:52:46.389409 (+1257675us) ts_tablet_manager.cc:1145] Deleting tablet 3d662e0ce13f447595e5f92b94de3018
0112 02:52:46.389419 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.389482 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1257518}
W20250112 02:52:46.392835 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 896908b2726c40f9aabdfc9a6b3283f7 consensus state has no leader
I20250112 02:52:46.393373 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 13578542e9e741e8817e0ee63b10f1a1 (table default.Table_63abe3ac694a4ed6bf4aef720b542e73 [id=a7ce19c248fe4ac6a6b436773d6c1b4c]) successfully deleted
I20250112 02:52:46.393318 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_f8ab3098ff7f4c479588888eac14e9a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.393833 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8ab3098ff7f4c479588888eac14e9a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.382964 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 61d7703b15024fe7a35805345d5e81ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:46.395602 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 3d662e0ce13f447595e5f92b94de3018 because the tablet was not found. No further retry: Not found: Tablet not found: 3d662e0ce13f447595e5f92b94de3018
I20250112 02:52:46.395766 3111 tablet_replica.cc:331] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:46.396495 3111 raft_consensus.cc:2238] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:46.398203 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:46.399843 3111 raft_consensus.cc:2267] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.399623 449 catalog_manager.cc:5526] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:46.400032 449 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:46.400555 3111 ts_tablet_manager.cc:1905] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.403257 449 catalog_manager.cc:5526] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:46.405479 449 catalog_manager.cc:5526] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:46.407359 453 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet f2d698838e574f259e669f2929a1f257
I20250112 02:52:46.409834 454 master_service.cc:946] ReplaceTablet: received request to replace tablet a1123077650c4e89bffddd6d1264e428 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:46.414817 2978 consensus_queue.cc:1035] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:46.416568 3167 tablet.cc:1721] T c73f2fa452b84be398476b8aba6c9a08 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:46.417832 3177 tablet.cc:1721] T c73f2fa452b84be398476b8aba6c9a08 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:46.438586 2978 consensus_queue.cc:1035] T 9bbef42e11a843b18ed582ab4d04a92a P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:46.438745 1177 tablet_bootstrap.cc:492] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:46.438874 1177 ts_tablet_manager.cc:1397] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.208s user 0.004s sys 0.000s
I20250112 02:52:46.439679 1177 raft_consensus.cc:357] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.441407 1173 tablet_bootstrap.cc:492] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:46.441545 1173 ts_tablet_manager.cc:1397] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.197s user 0.007s sys 0.023s
I20250112 02:52:46.442230 1173 raft_consensus.cc:357] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.442395 1173 raft_consensus.cc:383] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:46.442457 1173 raft_consensus.cc:738] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:46.442652 1173 consensus_queue.cc:260] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.443179 1173 ts_tablet_manager.cc:1428] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:46.443426 1173 tablet_bootstrap.cc:492] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:46.448783 2977 raft_consensus.cc:491] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:46.450357 2977 raft_consensus.cc:513] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.450940 2977 leader_election.cc:290] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:46.450196 3158 raft_consensus.cc:491] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:46.451251 3158 raft_consensus.cc:513] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.451823 3158 leader_election.cc:290] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:46.454669 1173 tablet_bootstrap.cc:654] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:46.463577 3111 ts_tablet_manager.cc:1918] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:46.363277 3000 raft_consensus.cc:491] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:46.464040 3000 raft_consensus.cc:513] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.464658 3000 leader_election.cc:290] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:46.464964 3000 raft_consensus.cc:491] T b114b98d1cf4419a8bfeef558d818c01 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:46.465046 3000 raft_consensus.cc:513] T b114b98d1cf4419a8bfeef558d818c01 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:46.465436 3000 leader_election.cc:290] T b114b98d1cf4419a8bfeef558d818c01 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:46.382265 855 leader_election.cc:343] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:46.465572 855 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 83309.5us
I20250112 02:52:46.465978 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b3c5a861a90840d0a33b780ec75bffd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.463788 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d47f6d663934b25a0200ee9fa81165d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:46.466297 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 5804fc4130f84c69832c1022b7bd7fa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.466445 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 61d7703b15024fe7a35805345d5e81ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.466493 3111 log.cc:1198] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/058a9ca089cd4967887b0a1127c7d03f
I20250112 02:52:46.417208 3183 raft_consensus.cc:491] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:46.466747 3111 ts_tablet_manager.cc:1939] T 058a9ca089cd4967887b0a1127c7d03f P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:46.466794 3183 raft_consensus.cc:513] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:46.467264 3183 leader_election.cc:290] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:46.467378 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 414) took 1337 ms. Trace:
I20250112 02:52:46.468405 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 3499374810fb412e938325434c35618d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.463232 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efcd32b4ccc4fada5adb3ca3bcdddaa" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:46.468852 918 raft_consensus.cc:2463] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:46.469897 2977 consensus_queue.cc:1035] T f69b7a0f52304ad9951a07b3ab0372e7 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:46.447677 3109 ts_tablet_manager.cc:1918] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:46.452636 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efcd32b4ccc4fada5adb3ca3bcdddaa" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:46.471093 991 leader_election.cc:343] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:46.452728 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e7882e07784bdda18dc1ffd20963c8" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:46.467875 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b114b98d1cf4419a8bfeef558d818c01" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:46.467942 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09900b42e8a443e680cc35c96a1dbbb7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:46.468487 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681a930541a84f0197ecafe670278e70" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:46.471632 991 leader_election.cc:343] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:46.471997 855 leader_election.cc:343] T b114b98d1cf4419a8bfeef558d818c01 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:46.472167 855 leader_election.cc:343] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:46.472285 855 leader_election.cc:343] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:46.472749 989 leader_election.cc:304] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:46.473147 3158 raft_consensus.cc:2798] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:46.473258 3158 raft_consensus.cc:695] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:46.473467 3158 consensus_queue.cc:237] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.474196 3109 log.cc:1198] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/13578542e9e741e8817e0ee63b10f1a1
I20250112 02:52:46.474682 3109 ts_tablet_manager.cc:1939] T 13578542e9e741e8817e0ee63b10f1a1 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:46.475721 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 429) took 1655 ms. Trace:
I20250112 02:52:46.475863 3109 rpcz_store.cc:276] 0112 02:52:44.820308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.820369 (+ 61us) service_pool.cc:225] Handling call
0112 02:52:46.383689 (+1563320us) ts_tablet_manager.cc:1145] Deleting tablet 13578542e9e741e8817e0ee63b10f1a1
0112 02:52:46.383699 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.388170 (+ 4471us) spinlock_profiling.cc:243] Waited 3.69 ms on lock 0x6160000c8554. stack: 00007f1bc077efd6 00007f1bc077ed90 00007f1bbf75e48d 000000000096d544 000000000096d50d 000000000096ceda 00007f1bcc021ed0 00007f1bcd316463 00007f1bcfdbaeb7 00007f1bcfded584 00007f1bcfdce690 00007f1bcfdce41d 0000000000a71ecc 00007f1bc07c5b64 00007f1bc07c918b 00007f1bc07c9008
0112 02:52:46.399240 (+ 11070us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.447586 (+ 48346us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.475699 (+ 28113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8489216,"tablet-delete.queue_time_us":1563170}
I20250112 02:52:46.476455 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 13578542e9e741e8817e0ee63b10f1a1 (table default.Table_63abe3ac694a4ed6bf4aef720b542e73 [id=a7ce19c248fe4ac6a6b436773d6c1b4c]) successfully deleted
I20250112 02:52:46.476557 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 430) took 1655 ms. Trace:
I20250112 02:52:46.476641 3109 rpcz_store.cc:276] 0112 02:52:44.820672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.820850 (+ 178us) service_pool.cc:225] Handling call
0112 02:52:46.476421 (+1655571us) ts_tablet_manager.cc:1145] Deleting tablet cbe30f9860e2492b90e8cedf8ff9daba
0112 02:52:46.476434 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.476540 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1655439}
W20250112 02:52:46.477047 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet cbe30f9860e2492b90e8cedf8ff9daba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbe30f9860e2492b90e8cedf8ff9daba already in progress: creating tablet
I20250112 02:52:46.477157 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 431) took 1656 ms. Trace:
I20250112 02:52:46.477252 3109 rpcz_store.cc:276] 0112 02:52:44.820794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.820989 (+ 195us) service_pool.cc:225] Handling call
0112 02:52:46.477051 (+1656062us) ts_tablet_manager.cc:1145] Deleting tablet 8c821ea3ca4946c290c17135366e0cac
0112 02:52:46.477062 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.477141 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1655910}
W20250112 02:52:46.477644 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 8c821ea3ca4946c290c17135366e0cac because the tablet was not found. No further retry: Not found: Tablet not found: 8c821ea3ca4946c290c17135366e0cac
I20250112 02:52:46.477597 3111 rpcz_store.cc:276] 0112 02:52:45.130375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.132418 (+ 2043us) service_pool.cc:225] Handling call
0112 02:52:46.395718 (+1263300us) ts_tablet_manager.cc:1145] Deleting tablet 058a9ca089cd4967887b0a1127c7d03f
0112 02:52:46.395729 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.448355 (+ 52626us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.463486 (+ 15131us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.467361 (+ 3875us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1263124}
I20250112 02:52:46.478070 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 058a9ca089cd4967887b0a1127c7d03f (table default.Table_a56064e2c6b449b7b1e32f2a69f2f7fe [id=448877a19b0d4ed2abbc3a796e40eaeb]) successfully deleted
I20250112 02:52:46.478297 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 428) took 1658 ms. Trace:
I20250112 02:52:46.478395 3109 rpcz_store.cc:276] 0112 02:52:44.819994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.820375 (+ 381us) service_pool.cc:225] Handling call
0112 02:52:46.478186 (+1657811us) ts_tablet_manager.cc:1145] Deleting tablet d0cb54003a83472cb9c57ace3e65b0be
0112 02:52:46.478194 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.478278 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1647712}
W20250112 02:52:46.478770 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d0cb54003a83472cb9c57ace3e65b0be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0cb54003a83472cb9c57ace3e65b0be already in progress: creating tablet
I20250112 02:52:46.479323 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 422) took 1311 ms. Trace:
I20250112 02:52:46.479418 3111 rpcz_store.cc:276] 0112 02:52:45.168283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.169027 (+ 744us) service_pool.cc:225] Handling call
0112 02:52:46.479198 (+1310171us) ts_tablet_manager.cc:1145] Deleting tablet 4cfe1a1595014c27bbd350cec169d71a
0112 02:52:46.479211 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.479305 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1309960}
W20250112 02:52:46.479827 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 4cfe1a1595014c27bbd350cec169d71a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cfe1a1595014c27bbd350cec169d71a already in progress: creating tablet
I20250112 02:52:46.482441 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 436) took 1651 ms. Trace:
I20250112 02:52:46.482540 3109 rpcz_store.cc:276] 0112 02:52:44.831140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.831182 (+ 42us) service_pool.cc:225] Handling call
0112 02:52:46.482333 (+1651151us) ts_tablet_manager.cc:1145] Deleting tablet 795ac7eaf81d48bb926865d9ebb388b6
0112 02:52:46.482353 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.482422 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1650993}
W20250112 02:52:46.482983 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 795ac7eaf81d48bb926865d9ebb388b6 because the tablet was not found. No further retry: Not found: Tablet not found: 795ac7eaf81d48bb926865d9ebb388b6
I20250112 02:52:46.483713 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 423) took 1278 ms. Trace:
I20250112 02:52:46.483820 3111 rpcz_store.cc:276] 0112 02:52:45.204938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.205745 (+ 807us) service_pool.cc:225] Handling call
0112 02:52:46.483587 (+1277842us) ts_tablet_manager.cc:1145] Deleting tablet ea33053c62dc4707b759d5c1756ab4bd
0112 02:52:46.483600 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.483693 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1277618}
I20250112 02:52:46.484714 3109 tablet_replica.cc:331] T 54215c55966a46e1ab96e352c4e3764e P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:46.484995 3109 raft_consensus.cc:2238] T 54215c55966a46e1ab96e352c4e3764e P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:46.485165 3109 raft_consensus.cc:2267] T 54215c55966a46e1ab96e352c4e3764e P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:46.485582 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 85634.8us
I20250112 02:52:46.486075 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 424) took 1280 ms. Trace:
I20250112 02:52:46.486153 3111 rpcz_store.cc:276] 0112 02:52:45.207806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.207850 (+ 44us) service_pool.cc:225] Handling call
0112 02:52:46.485983 (+1278133us) ts_tablet_manager.cc:1145] Deleting tablet a4328137407843e1879493ef0c56ffc8
0112 02:52:46.485992 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.486060 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1277955}
I20250112 02:52:46.487352 3174 consensus_queue.cc:1035] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:46.470661 1177 raft_consensus.cc:383] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:46.490999 3159 consensus_queue.cc:1035] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:46.491201 1177 raft_consensus.cc:738] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:46.491509 1177 consensus_queue.cc:260] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.494344 895 tablet_service.cc:1514] Processing DeleteTablet for tablet ea33053c62dc4707b759d5c1756ab4bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.494575 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 76009bac22ce4136b148b493a004a0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.497421 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 425) took 1278 ms. Trace:
I20250112 02:52:46.497699 898 tablet_service.cc:1514] Processing DeleteTablet for tablet cbe30f9860e2492b90e8cedf8ff9daba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.497782 3111 rpcz_store.cc:276] 0112 02:52:45.208136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.208168 (+ 32us) service_pool.cc:225] Handling call
0112 02:52:46.486348 (+1278180us) ts_tablet_manager.cc:1145] Deleting tablet f52b838214c54ac69ad30dac6e56669a
0112 02:52:46.486355 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.486416 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1278052}
I20250112 02:52:46.498034 3111 tablet_replica.cc:331] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:46.498189 3111 raft_consensus.cc:2238] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:46.498361 3111 pending_rounds.cc:62] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6: Trying to abort 1 pending ops.
I20250112 02:52:46.498404 3111 pending_rounds.cc:69] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319901077532672 op_type: NO_OP noop_request { }
I20250112 02:52:46.498476 3111 raft_consensus.cc:2883] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:46.498534 3111 raft_consensus.cc:2267] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.498643 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d662e0ce13f447595e5f92b94de3018 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:44 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.499408 3109 ts_tablet_manager.cc:1905] T 54215c55966a46e1ab96e352c4e3764e P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.501123 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_5d4ae07cad674fa4a7d2253d5eccaa5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.501674 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d4ae07cad674fa4a7d2253d5eccaa5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.502269 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_93c04d06452c40b680ed534a136f7eb8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.502732 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93c04d06452c40b680ed534a136f7eb8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.502893 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_7751762ace034e18aa56758a720947a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.503324 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7751762ace034e18aa56758a720947a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.505409 3111 ts_tablet_manager.cc:1905] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.506580 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_6e77252e817f43399c9b3f5b221795ad" } new_table_name: "default.Table_97a56432deb04f0ea0d6f838de79af39" modify_external_catalogs: true
I20250112 02:52:46.507994 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_ee86b03e71a1459f8fbf833684f9b321"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.508394 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee86b03e71a1459f8fbf833684f9b321 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.510298 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 37d38f5e525440f8a0f7822620933435 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:46.515841 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 681a930541a84f0197ecafe670278e70 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b1d46c5395d4defb2d0e2948807eaf4) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.516021 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cfe1a1595014c27bbd350cec169d71a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.516211 898 tablet_service.cc:1514] Processing DeleteTablet for tablet a4328137407843e1879493ef0c56ffc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.516319 895 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cb54003a83472cb9c57ace3e65b0be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:40 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.517674 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 93aba9b4d1dd485cb62c4827c916a1d5 with delete_type TABLET_DATA_DELETED (replaced by tablet e3454657cbd74544b0ae962736e4b7b5) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.493492 1177 ts_tablet_manager.cc:1428] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.055s user 0.002s sys 0.000s
I20250112 02:52:46.518126 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 940affdc2c484b7d9fd2de17f9d1cea7 with delete_type TABLET_DATA_DELETED (replaced by tablet ab183c43f6a64270ad744e61364f0a99) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.518208 887 tablet_service.cc:1514] Processing DeleteTablet for tablet f3771166b77e4f90aaa116ab1a9dc299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:39 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.518330 898 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e7882e07784bdda18dc1ffd20963c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.518740 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 76009bac22ce4136b148b493a004a0f7 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:46.519272 1177 tablet_bootstrap.cc:492] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
W20250112 02:52:46.519701 989 leader_election.cc:343] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: a8e7882e07784bdda18dc1ffd20963c8
I20250112 02:52:46.519889 989 leader_election.cc:304] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:46.522517 1177 tablet_bootstrap.cc:654] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:46.523087 451 master_service.cc:946] ReplaceTablet: received request to replace tablet f5c671e327a148ecb1d08a983448946e from {username='slave'} at 127.0.0.1:40610
W20250112 02:52:46.526059 854 leader_election.cc:343] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:46.526643 3110 ts_tablet_manager.cc:1918] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:46.528687 3110 log.cc:1198] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/f52b838214c54ac69ad30dac6e56669a
I20250112 02:52:46.529439 3110 ts_tablet_manager.cc:1939] T f52b838214c54ac69ad30dac6e56669a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:46.530072 3111 ts_tablet_manager.cc:1918] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:46.527401 854 leader_election.cc:304] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:46.532297 3183 raft_consensus.cc:2743] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:46.534631 1173 tablet_bootstrap.cc:492] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:46.534878 1173 ts_tablet_manager.cc:1397] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.092s user 0.000s sys 0.009s
I20250112 02:52:46.535598 1173 raft_consensus.cc:357] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.535756 1173 raft_consensus.cc:383] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:46.535807 1173 raft_consensus.cc:738] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:46.536007 1173 consensus_queue.cc:260] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.536475 1173 ts_tablet_manager.cc:1428] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 02:52:46.536727 1173 tablet_bootstrap.cc:492] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:46.537578 3159 raft_consensus.cc:2743] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:46.538302 1173 tablet_bootstrap.cc:654] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:46.539104 3111 log.cc:1198] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/d95f98ae7bdf42518bbff55e257fea88
I20250112 02:52:46.539520 3111 ts_tablet_manager.cc:1939] T d95f98ae7bdf42518bbff55e257fea88 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:46.541250 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet a1123077650c4e89bffddd6d1264e428
I20250112 02:52:46.541719 454 catalog_manager.cc:6596] ReplaceTablet: tablet a1123077650c4e89bffddd6d1264e428 deleted and replaced by tablet 08c1a4ccdb664c05a9999a9843ded765
I20250112 02:52:46.545161 898 tablet_service.cc:1514] Processing DeleteTablet for tablet a1123077650c4e89bffddd6d1264e428 with delete_type TABLET_DATA_DELETED (replaced by tablet 08c1a4ccdb664c05a9999a9843ded765) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.545373 1173 tablet_bootstrap.cc:492] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:46.545517 1173 ts_tablet_manager.cc:1397] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20250112 02:52:46.546178 1173 raft_consensus.cc:357] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.546392 1173 raft_consensus.cc:383] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:46.546496 1173 raft_consensus.cc:738] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:46.547580 1173 consensus_queue.cc:260] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.548251 1173 ts_tablet_manager.cc:1428] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20250112 02:52:46.549906 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a470f84d50df4cf0b4a1e5d248b4c32a consensus state has no leader
I20250112 02:52:46.550128 989 leader_election.cc:304] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:46.551424 3266 raft_consensus.cc:2798] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:46.551568 3266 raft_consensus.cc:491] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:46.551843 3266 raft_consensus.cc:3054] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:46.552315 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09900b42e8a443e680cc35c96a1dbbb7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:46.552443 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b114b98d1cf4419a8bfeef558d818c01" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:46.552503 1055 raft_consensus.cc:2463] T 09900b42e8a443e680cc35c96a1dbbb7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
W20250112 02:52:46.553213 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f52b838214c54ac69ad30dac6e56669a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f52b838214c54ac69ad30dac6e56669a already in progress: creating tablet
I20250112 02:52:46.553285 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681a930541a84f0197ecafe670278e70" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:46.556491 1173 tablet_bootstrap.cc:492] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:46.558315 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet a1123077650c4e89bffddd6d1264e428 with delete_type TABLET_DATA_DELETED (replaced by tablet 08c1a4ccdb664c05a9999a9843ded765) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.561491 1173 tablet_bootstrap.cc:654] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:46.563169 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 426) took 1343 ms. Trace:
I20250112 02:52:46.563284 3111 rpcz_store.cc:276] 0112 02:52:45.208438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.208473 (+ 35us) service_pool.cc:225] Handling call
0112 02:52:46.497994 (+1289521us) ts_tablet_manager.cc:1145] Deleting tablet d95f98ae7bdf42518bbff55e257fea88
0112 02:52:46.498002 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.510996 (+ 12994us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.529992 (+ 18996us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.551800 (+ 21808us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1289377}
I20250112 02:52:46.564069 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 427) took 1355 ms. Trace:
I20250112 02:52:46.564172 3111 rpcz_store.cc:276] 0112 02:52:45.208726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.208760 (+ 34us) service_pool.cc:225] Handling call
0112 02:52:46.563953 (+1355193us) ts_tablet_manager.cc:1145] Deleting tablet 64b0d58af4cf4adcbeb3ef5070f2982d
0112 02:52:46.563962 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.564051 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1354951}
I20250112 02:52:46.564534 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 428) took 1321 ms. Trace:
I20250112 02:52:46.564630 3111 rpcz_store.cc:276] 0112 02:52:45.242601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.242789 (+ 188us) service_pool.cc:225] Handling call
0112 02:52:46.564437 (+1321648us) ts_tablet_manager.cc:1145] Deleting tablet be856e4b913e40d58a3f3ba54698cc5f
0112 02:52:46.564446 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.564519 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1321437}
I20250112 02:52:46.565943 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet f5c671e327a148ecb1d08a983448946e
I20250112 02:52:46.566648 451 catalog_manager.cc:6596] ReplaceTablet: tablet f5c671e327a148ecb1d08a983448946e deleted and replaced by tablet e438e49f1e5345a78e11b66ac8e95d28
I20250112 02:52:46.569413 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet d95f98ae7bdf42518bbff55e257fea88 (table default.Table_2fcdbb05970f4a098f7ba3e287fa0348 [id=70c61753e9944e1fbd65e9ec41a48ee2]) successfully deleted
I20250112 02:52:46.570384 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c671e327a148ecb1d08a983448946e with delete_type TABLET_DATA_DELETED (replaced by tablet e438e49f1e5345a78e11b66ac8e95d28) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.572129 3111 tablet_replica.cc:331] T 26577b614e32458e8bc5d6830c8c2e40 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
W20250112 02:52:46.572399 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet be856e4b913e40d58a3f3ba54698cc5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be856e4b913e40d58a3f3ba54698cc5f already in progress: creating tablet
I20250112 02:52:46.575646 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 99407ead9cf547e3bbe78c0d0b327e6a from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:46.575711 3111 raft_consensus.cc:2238] T 26577b614e32458e8bc5d6830c8c2e40 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:46.575971 3111 raft_consensus.cc:2267] T 26577b614e32458e8bc5d6830c8c2e40 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.576035 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 353) took 1394 ms. Trace:
I20250112 02:52:46.576169 3110 rpcz_store.cc:276] 0112 02:52:45.181642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.182128 (+ 486us) service_pool.cc:225] Handling call
0112 02:52:46.293599 (+1111471us) ts_tablet_manager.cc:1145] Deleting tablet f52b838214c54ac69ad30dac6e56669a
0112 02:52:46.293608 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.300511 (+ 6903us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.472367 (+171856us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.576013 (+103646us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1111337}
I20250112 02:52:46.576423 898 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c671e327a148ecb1d08a983448946e with delete_type TABLET_DATA_DELETED (replaced by tablet e438e49f1e5345a78e11b66ac8e95d28) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.576853 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet f52b838214c54ac69ad30dac6e56669a (table default.Table_2989b0fb83514c45ac9836e95e444f31 [id=f612d417385a4a02814d993aab77b5a5]) successfully deleted
I20250112 02:52:46.577157 3110 tablet_replica.cc:331] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:46.577392 3110 raft_consensus.cc:2238] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:46.577586 3110 raft_consensus.cc:2267] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.575371 3266 raft_consensus.cc:513] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.578150 3110 ts_tablet_manager.cc:1905] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.578738 3266 leader_election.cc:290] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:46.579165 3111 ts_tablet_manager.cc:1905] T 26577b614e32458e8bc5d6830c8c2e40 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.534719 3109 ts_tablet_manager.cc:1918] T 54215c55966a46e1ab96e352c4e3764e P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:46.581535 3109 log.cc:1198] T 54215c55966a46e1ab96e352c4e3764e P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/54215c55966a46e1ab96e352c4e3764e
I20250112 02:52:46.543192 746 tablet_service.cc:1514] Processing DeleteTablet for tablet a1123077650c4e89bffddd6d1264e428 with delete_type TABLET_DATA_DELETED (replaced by tablet 08c1a4ccdb664c05a9999a9843ded765) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:46.567911 747 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c671e327a148ecb1d08a983448946e with delete_type TABLET_DATA_DELETED (replaced by tablet e438e49f1e5345a78e11b66ac8e95d28) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:46.573889 3138 consensus_queue.cc:1035] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:46.578702 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efcd32b4ccc4fada5adb3ca3bcdddaa" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:46.581939 3109 ts_tablet_manager.cc:1939] T 54215c55966a46e1ab96e352c4e3764e P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
W20250112 02:52:46.583931 991 leader_election.cc:343] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:46.584439 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 438) took 1753 ms. Trace:
I20250112 02:52:46.584506 3109 rpcz_store.cc:276] 0112 02:52:44.831311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.831347 (+ 36us) service_pool.cc:225] Handling call
0112 02:52:46.484663 (+1653316us) ts_tablet_manager.cc:1145] Deleting tablet 54215c55966a46e1ab96e352c4e3764e
0112 02:52:46.484675 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.523174 (+ 38499us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.534031 (+ 10857us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.584422 (+ 50391us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1653196}
I20250112 02:52:46.584834 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 435) took 1753 ms. Trace:
I20250112 02:52:46.584894 3109 rpcz_store.cc:276] 0112 02:52:44.831062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.831516 (+ 454us) service_pool.cc:225] Handling call
0112 02:52:46.584728 (+1753212us) ts_tablet_manager.cc:1145] Deleting tablet 8a88463991294bd8a335337f8b072c3d
0112 02:52:46.584736 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.584821 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1753087}
I20250112 02:52:46.585111 3109 tablet_replica.cc:331] T 058a9ca089cd4967887b0a1127c7d03f P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:46.585275 3109 raft_consensus.cc:2238] T 058a9ca089cd4967887b0a1127c7d03f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:46.585398 3109 raft_consensus.cc:2267] T 058a9ca089cd4967887b0a1127c7d03f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.585779 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 54215c55966a46e1ab96e352c4e3764e (table default.Table_97a56432deb04f0ea0d6f838de79af39 [id=1fa1474ab8804667b22e5f30c48c1630]) successfully deleted
W20250112 02:52:46.586048 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 8a88463991294bd8a335337f8b072c3d because the tablet was not found. No further retry: Not found: Tablet not found: 8a88463991294bd8a335337f8b072c3d
I20250112 02:52:46.587952 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_e495ee84abe3479c80c8ab2e14f86c96" } modify_external_catalogs: true
I20250112 02:52:46.588053 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_3185108f36ed429e9a9967cc2b11072e" } new_table_name: "default.Table_5919a70d0f014cf2b979fe6a6810b735" modify_external_catalogs: true
I20250112 02:52:46.588163 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_e495ee84abe3479c80c8ab2e14f86c96" } modify_external_catalogs: true
I20250112 02:52:46.589658 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_46a03fa2d75e4541a98a1b3d91f85395" } new_table_name: "default.Table_16bf9ce62e194344833127d6803825f2" modify_external_catalogs: true
I20250112 02:52:46.589807 776 raft_consensus.cc:3054] T 125b12a2f9164653be79d3e6d8313d36 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:46.590761 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_29f995195bb64f709d7b2eee97f35454" } modify_external_catalogs: true
I20250112 02:52:46.590910 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_29f995195bb64f709d7b2eee97f35454" } modify_external_catalogs: true
I20250112 02:52:46.591874 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_22969d80f802435898d7d60983f0daf9" } modify_external_catalogs: true
I20250112 02:52:46.592015 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_22969d80f802435898d7d60983f0daf9" } modify_external_catalogs: true
I20250112 02:52:46.592476 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_8ab566cf9c8f45eba4dc9bb85264c823" } modify_external_catalogs: true
I20250112 02:52:46.592625 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_8ab566cf9c8f45eba4dc9bb85264c823" } modify_external_catalogs: true
I20250112 02:52:46.592887 3158 consensus_queue.cc:1035] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:46.593633 3109 ts_tablet_manager.cc:1905] T 058a9ca089cd4967887b0a1127c7d03f P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.600392 3109 ts_tablet_manager.cc:1918] T 058a9ca089cd4967887b0a1127c7d03f P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:46.600715 3109 log.cc:1198] T 058a9ca089cd4967887b0a1127c7d03f P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/058a9ca089cd4967887b0a1127c7d03f
I20250112 02:52:46.601125 3109 ts_tablet_manager.cc:1939] T 058a9ca089cd4967887b0a1127c7d03f P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:46.601670 3175 raft_consensus.cc:491] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:46.601799 3175 raft_consensus.cc:513] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.602919 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500f0c5c76e54cd8808e6c8611e99d83" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:46.603112 1055 raft_consensus.cc:2463] T 500f0c5c76e54cd8808e6c8611e99d83 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:46.603505 714 leader_election.cc:304] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:46.604840 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_2e65b56ef2f9458ea5125a6f045569a6" } new_table_name: "default.Table_fb9557bec89748dabf949080f016deba" modify_external_catalogs: true
W20250112 02:52:46.605185 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cca2a096fce0464394f5029652813873 consensus state has no leader
I20250112 02:52:46.606369 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet e477fc157aed4563923d85d66014293a
I20250112 02:52:46.610780 3221 tablet.cc:1721] T c73f2fa452b84be398476b8aba6c9a08 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:46.612593 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet e477fc157aed4563923d85d66014293a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.613607 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_a0ccfcee886d449d8bfd5bc1f9ca38df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.614104 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0ccfcee886d449d8bfd5bc1f9ca38df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.614864 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efcd32b4ccc4fada5adb3ca3bcdddaa" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:46.615051 917 raft_consensus.cc:3054] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:46.615177 898 tablet_service.cc:1514] Processing DeleteTablet for tablet e477fc157aed4563923d85d66014293a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.615250 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_7751762ace034e18aa56758a720947a4" } new_table_name: "default.Table_866e2b7327c64126af36d106ef79ce17" modify_external_catalogs: true
I20250112 02:52:46.616355 854 leader_election.cc:304] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:46.616652 3260 raft_consensus.cc:2798] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:46.616644 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_01fbb219edf24a889c196cc25eafc530"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:46.616726 3260 raft_consensus.cc:491] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:46.616791 3260 raft_consensus.cc:3054] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:46.617053 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01fbb219edf24a889c196cc25eafc530 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:46.617549 854 leader_election.cc:343] T b114b98d1cf4419a8bfeef558d818c01 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:46.617703 854 leader_election.cc:304] T b114b98d1cf4419a8bfeef558d818c01 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:46.618100 854 leader_election.cc:343] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:46.618263 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 48671.5us
I20250112 02:52:46.618736 854 leader_election.cc:304] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:46.621932 1173 tablet_bootstrap.cc:492] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:46.622089 1173 ts_tablet_manager.cc:1397] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.001s
I20250112 02:52:46.623032 3000 raft_consensus.cc:2743] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:46.623212 3000 raft_consensus.cc:491] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:46.623250 1173 raft_consensus.cc:357] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.623296 3000 raft_consensus.cc:513] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:46.623430 1173 raft_consensus.cc:383] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:46.623522 1173 raft_consensus.cc:738] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:46.619438 3260 raft_consensus.cc:513] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.623927 1173 consensus_queue.cc:260] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.624445 1173 ts_tablet_manager.cc:1428] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 02:52:46.624480 3260 leader_election.cc:290] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:46.625027 446 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 136c7f767fb74c2aa961d420f77c98dd
I20250112 02:52:46.625491 1173 tablet_bootstrap.cc:492] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:46.625571 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bbaa7dfc7da14cc2b4f3731d3da6107b has no consensus state
I20250112 02:52:46.625846 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_e82fa0f576134ad087c035f9516863a3" } modify_external_catalogs: true
I20250112 02:52:46.626003 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_e82fa0f576134ad087c035f9516863a3" } modify_external_catalogs: true
I20250112 02:52:46.626782 1173 tablet_bootstrap.cc:654] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:46.626914 3262 raft_consensus.cc:2743] T b114b98d1cf4419a8bfeef558d818c01 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:46.625483 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_bb6854d0dab24b32a42819a3c18ac0d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:46.627430 3000 leader_election.cc:290] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:46.627663 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb6854d0dab24b32a42819a3c18ac0d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.628288 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09900b42e8a443e680cc35c96a1dbbb7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:46.628423 1055 raft_consensus.cc:3054] T 09900b42e8a443e680cc35c96a1dbbb7 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:46.628518 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cb54003a83472cb9c57ace3e65b0be" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:46.630448 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 434) took 1771 ms. Trace:
W20250112 02:52:46.630779 854 leader_election.cc:343] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:46.630861 1055 raft_consensus.cc:2463] T 09900b42e8a443e680cc35c96a1dbbb7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 1.
I20250112 02:52:46.633545 917 raft_consensus.cc:2463] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:46.634609 854 leader_election.cc:304] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:46.634840 3000 raft_consensus.cc:2798] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:46.634925 989 leader_election.cc:304] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:46.634971 3000 raft_consensus.cc:695] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Becoming Leader. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Running, Role: LEADER
I20250112 02:52:46.635367 3159 raft_consensus.cc:2798] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:46.635488 3159 raft_consensus.cc:695] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:46.635680 3159 consensus_queue.cc:237] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.636209 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500f0c5c76e54cd8808e6c8611e99d83" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:46.636420 914 raft_consensus.cc:2463] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:46.635224 3000 consensus_queue.cc:237] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.645184 3111 ts_tablet_manager.cc:1918] T 26577b614e32458e8bc5d6830c8c2e40 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:46.645298 3111 log.cc:1198] T 26577b614e32458e8bc5d6830c8c2e40 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/26577b614e32458e8bc5d6830c8c2e40
I20250112 02:52:46.645673 3111 ts_tablet_manager.cc:1939] T 26577b614e32458e8bc5d6830c8c2e40 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:46.646369 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 434) took 1245 ms. Trace:
I20250112 02:52:46.646476 3111 rpcz_store.cc:276] 0112 02:52:45.400597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.400648 (+ 51us) service_pool.cc:225] Handling call
0112 02:52:46.572080 (+1171432us) ts_tablet_manager.cc:1145] Deleting tablet 26577b614e32458e8bc5d6830c8c2e40
0112 02:52:46.572091 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.581883 (+ 9792us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.638566 (+ 56683us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.646352 (+ 7786us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1171249}
I20250112 02:52:46.646816 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 26577b614e32458e8bc5d6830c8c2e40 (table default.Table_03e3448d809c4fc792d0bff94a713044 [id=5f3cc73b47414c438591107e35a12472]) successfully deleted
W20250112 02:52:46.647979 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2e2a64032f84fadb05f4455c2c51529 has no consensus state
I20250112 02:52:46.649960 1173 tablet_bootstrap.cc:492] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:46.650146 1173 ts_tablet_manager.cc:1397] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.002s
I20250112 02:52:46.650831 1173 raft_consensus.cc:357] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.654384 3175 leader_election.cc:290] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:46.651072 1173 raft_consensus.cc:383] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:46.655280 1173 raft_consensus.cc:738] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:46.655467 3138 raft_consensus.cc:2798] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:46.655680 3138 raft_consensus.cc:491] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:46.655783 3138 raft_consensus.cc:3054] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:46.655680 1173 consensus_queue.cc:260] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.657219 1173 ts_tablet_manager.cc:1428] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250112 02:52:46.658375 3138 raft_consensus.cc:513] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.658699 1173 tablet_bootstrap.cc:492] T 3499374810fb412e938325434c35618d P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:46.655542 746 tablet_service.cc:1514] Processing DeleteTablet for tablet e477fc157aed4563923d85d66014293a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:46.655560 714 consensus_peers.cc:487] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 896908b2726c40f9aabdfc9a6b3283f7. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:46.659060 452 catalog_manager.cc:5526] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:46.659493 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_a9907ed8bfe14625977ec98b67e9d599 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 2 got 0
I20250112 02:52:46.660185 898 tablet_service.cc:1467] Processing CreateTablet for tablet f810f10151c84e648b338a55fb181277 (DEFAULT_TABLE table=default.Table_745092ddecfa47d3a2772ff78e6eab05 [id=b6ffbb39eb124fba8a5c4d048613acfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.660079 452 catalog_manager.cc:5526] T 2d3d291ca3494f05ac23c7fe293e0bd9 P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:46.660230 746 tablet_service.cc:1467] Processing CreateTablet for tablet f810f10151c84e648b338a55fb181277 (DEFAULT_TABLE table=default.Table_745092ddecfa47d3a2772ff78e6eab05 [id=b6ffbb39eb124fba8a5c4d048613acfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.660580 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f810f10151c84e648b338a55fb181277. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.660605 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f810f10151c84e648b338a55fb181277. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.660583 452 catalog_manager.cc:5526] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:46.660801 887 tablet_service.cc:1467] Processing CreateTablet for tablet fdc5a66017a54340a5c73e04b8ea95ba (DEFAULT_TABLE table=default.Table_4179e7e64aa24195973d2abac70761d4 [id=f14a433f942b4b5f8b824313cdaf7640]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.660905 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 05000c6a48e0450d8e6685ca864f67f0 (table default.Table_dcdbe8e16ced4132a8b540781fd01d2e [id=1ec97dc2b50a4a3dbbc449559938cead]). Expected version 1 got 0
I20250112 02:52:46.661094 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdc5a66017a54340a5c73e04b8ea95ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.661186 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:46.659680 1022 tablet_service.cc:1467] Processing CreateTablet for tablet f810f10151c84e648b338a55fb181277 (DEFAULT_TABLE table=default.Table_745092ddecfa47d3a2772ff78e6eab05 [id=b6ffbb39eb124fba8a5c4d048613acfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.661381 1028 tablet_service.cc:1467] Processing CreateTablet for tablet fdc5a66017a54340a5c73e04b8ea95ba (DEFAULT_TABLE table=default.Table_4179e7e64aa24195973d2abac70761d4 [id=f14a433f942b4b5f8b824313cdaf7640]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.661558 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c73f2fa452b84be398476b8aba6c9a08 (table default.Table_0c453685abe143aa82c64b441ec582bb [id=097d902a92934882bc7c016da518642a]). Expected version 3 got 2
I20250112 02:52:46.661607 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f810f10151c84e648b338a55fb181277. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.661772 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdc5a66017a54340a5c73e04b8ea95ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.661816 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:46.661975 452 catalog_manager.cc:5526] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:46.662312 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_9f558b53671c40849c3fe1c50dba42d8 [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 3 got 0
I20250112 02:52:46.662482 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_ee16ba338f554f4dbd75b332adf897f3 [id=f95ac050a4b840a19a8175caade87d48]). Expected version 2 got 0
I20250112 02:52:46.662695 452 catalog_manager.cc:5526] T c5079f87652a4ea58b366d1f9448fd4e P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:46.663036 452 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:46.664674 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09900b42e8a443e680cc35c96a1dbbb7" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:46.665014 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cb54003a83472cb9c57ace3e65b0be" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:46.665310 3106 raft_consensus.cc:491] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:46.665432 3106 raft_consensus.cc:513] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.665753 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_ac4a10685c414bef86c147c336335ff7 [id=78ee5f3a07714849a10192e80c425264] alter complete (version 2)
I20250112 02:52:46.666102 3106 leader_election.cc:290] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:46.666467 3138 leader_election.cc:290] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:46.667081 747 tablet_service.cc:1467] Processing CreateTablet for tablet fdc5a66017a54340a5c73e04b8ea95ba (DEFAULT_TABLE table=default.Table_4179e7e64aa24195973d2abac70761d4 [id=f14a433f942b4b5f8b824313cdaf7640]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.667428 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdc5a66017a54340a5c73e04b8ea95ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.668323 895 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4065e52d9643b1a1c530a8bdf4bfcc (DEFAULT_TABLE table=default.Table_327f33bee2d64c39a26bbcc21fe16736 [id=e3dc776d86c64291860484c4069f6e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.668663 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4065e52d9643b1a1c530a8bdf4bfcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.669471 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4065e52d9643b1a1c530a8bdf4bfcc (DEFAULT_TABLE table=default.Table_327f33bee2d64c39a26bbcc21fe16736 [id=e3dc776d86c64291860484c4069f6e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.669831 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4065e52d9643b1a1c530a8bdf4bfcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.670679 3111 tablet_replica.cc:331] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:46.670961 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c29a9ca43e034b43a00207e5847bbee3" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:46.671121 1053 raft_consensus.cc:2463] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:46.671103 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500f0c5c76e54cd8808e6c8611e99d83" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:46.671218 1050 raft_consensus.cc:3054] T 500f0c5c76e54cd8808e6c8611e99d83 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:46.671576 855 leader_election.cc:343] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:46.671883 855 leader_election.cc:343] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:46.671967 855 leader_election.cc:304] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:46.672354 3262 raft_consensus.cc:2743] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:46.673368 1050 raft_consensus.cc:2463] T 500f0c5c76e54cd8808e6c8611e99d83 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:46.674556 3111 raft_consensus.cc:2238] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:46.675525 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500f0c5c76e54cd8808e6c8611e99d83" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:46.675896 914 raft_consensus.cc:3054] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:46.676436 3111 raft_consensus.cc:2267] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.677814 1173 tablet_bootstrap.cc:654] T 3499374810fb412e938325434c35618d P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:46.678179 1022 tablet_service.cc:1467] Processing CreateTablet for tablet a66705bc87074cc8a661f351a6045e99 (DEFAULT_TABLE table=default.Table_21743df183e844c9a6ca2031b2bc81bc [id=ecf1b3d6636f4f6b834aa4b2961146ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.678570 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66705bc87074cc8a661f351a6045e99. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:46.680030 713 leader_election.cc:343] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: c29a9ca43e034b43a00207e5847bbee3
I20250112 02:52:46.680873 714 leader_election.cc:304] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:46.681027 891 tablet_service.cc:1467] Processing CreateTablet for tablet a66705bc87074cc8a661f351a6045e99 (DEFAULT_TABLE table=default.Table_21743df183e844c9a6ca2031b2bc81bc [id=ecf1b3d6636f4f6b834aa4b2961146ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.681216 3138 raft_consensus.cc:2798] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:46.681281 714 leader_election.cc:304] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters:
I20250112 02:52:46.681349 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66705bc87074cc8a661f351a6045e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.681944 760 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4065e52d9643b1a1c530a8bdf4bfcc (DEFAULT_TABLE table=default.Table_327f33bee2d64c39a26bbcc21fe16736 [id=e3dc776d86c64291860484c4069f6e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.682235 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4065e52d9643b1a1c530a8bdf4bfcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.687675 3245 ts_tablet_manager.cc:927] T ca6f080e5fd84084b601b646d4e6c9e2 P f1e775296eb746db8b318f17d26820e6: Initiating tablet copy from peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:46.687778 743 tablet_service.cc:1467] Processing CreateTablet for tablet a66705bc87074cc8a661f351a6045e99 (DEFAULT_TABLE table=default.Table_21743df183e844c9a6ca2031b2bc81bc [id=ecf1b3d6636f4f6b834aa4b2961146ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.687853 3111 ts_tablet_manager.cc:1905] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.688107 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66705bc87074cc8a661f351a6045e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.691877 3175 raft_consensus.cc:2798] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:46.692035 3175 raft_consensus.cc:695] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:46.692327 3175 consensus_queue.cc:237] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.681318 3138 raft_consensus.cc:491] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:46.703634 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 8c12a1c5e3584450ac193743a0107e80
I20250112 02:52:46.704706 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 17c54c5dc86446cf85974777c8145ef6
I20250112 02:52:46.705258 914 raft_consensus.cc:2463] T 500f0c5c76e54cd8808e6c8611e99d83 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:46.705765 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 17c54c5dc86446cf85974777c8145ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.707567 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 8fe57d374eb44c3fb1532484e214b654
I20250112 02:52:46.703169 3109 rpcz_store.cc:276] 0112 02:52:44.830989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.832530 (+ 1541us) service_pool.cc:225] Handling call
0112 02:52:46.585060 (+1752530us) ts_tablet_manager.cc:1145] Deleting tablet 058a9ca089cd4967887b0a1127c7d03f
0112 02:52:46.585068 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.597166 (+ 12098us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.600310 (+ 3144us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.602102 (+ 1792us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1752369}
I20250112 02:52:46.708647 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c12a1c5e3584450ac193743a0107e80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.709174 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe57d374eb44c3fb1532484e214b654 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.709837 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 99407ead9cf547e3bbe78c0d0b327e6a
I20250112 02:52:46.710347 451 catalog_manager.cc:6596] ReplaceTablet: tablet 99407ead9cf547e3bbe78c0d0b327e6a deleted and replaced by tablet a94cec0f0d4d4f06aa5667135efbf5a6
I20250112 02:52:46.710577 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe57d374eb44c3fb1532484e214b654 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.711252 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 99407ead9cf547e3bbe78c0d0b327e6a with delete_type TABLET_DATA_DELETED (replaced by tablet a94cec0f0d4d4f06aa5667135efbf5a6) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.711372 3138 raft_consensus.cc:3054] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:46.713233 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 99407ead9cf547e3bbe78c0d0b327e6a with delete_type TABLET_DATA_DELETED (replaced by tablet a94cec0f0d4d4f06aa5667135efbf5a6) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.713522 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_b0b0d76c3b4a428f9ffd93ebb6e96770 [id=ad0be37e6b6f4b9597dcb4cbd50d5b12] alter complete (version 4)
I20250112 02:52:46.716284 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 1d909275d67c4127b3144de53efc7d3e (DEFAULT_TABLE table=default.Table_ade612ff606d477c84a3de161a684cee [id=97365c7c50c94b049acbe232693c3543]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.716668 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d909275d67c4127b3144de53efc7d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.717434 3282 ts_tablet_manager.cc:927] T 164f76edcf8f4bd498dcd2c25fd7477d P f1e775296eb746db8b318f17d26820e6: Initiating tablet copy from peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:46.717384 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_b4223b0bf2f348158ec6104d6da3ba42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.717869 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4223b0bf2f348158ec6104d6da3ba42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.721247 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_542b9b0705ee44849946cbbb857a5f9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.721674 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_542b9b0705ee44849946cbbb857a5f9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.724963 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_ebbae2e0c50f44fd9e379196c2be90af [id=5c4769bfcefd4fe0a0f3c2e315d50c32] alter complete (version 7)
I20250112 02:52:46.725390 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_630d49265f684b0dbef983cb09ea0a14 [id=2c077da077834ae4bcc84efdd9e89120] alter complete (version 2)
I20250112 02:52:46.727192 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c12a1c5e3584450ac193743a0107e80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.728320 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_e44403a267f048b6aefc3ae0848fbab8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.728829 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e44403a267f048b6aefc3ae0848fbab8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.731401 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 17c54c5dc86446cf85974777c8145ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.734037 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_ebadf77c864c4883a04b9f62f15d7f49 [id=5cec72cf833347f3bfbca061f8feca41] alter complete (version 1)
I20250112 02:52:46.734339 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_16821effaf954ac28a375fb933aef7d3 [id=2c0e94e8c5774da1b8130c7651d94500] alter complete (version 2)
I20250112 02:52:46.737490 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d411f906b4d4eb68343bdcc493e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:33 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.737392 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f4bd2df0ce45dabc55373f1154d3ac with delete_type TABLET_DATA_DELETED (replaced by tablet f559dedfb7854cf890bef362c50b4213) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.737895 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 54215c55966a46e1ab96e352c4e3764e with delete_type TABLET_DATA_DELETED (replaced by tablet a470f84d50df4cf0b4a1e5d248b4c32a) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.739127 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 13578542e9e741e8817e0ee63b10f1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.741329 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 81c56c2cd7fa45f390602adb67681f04 (DEFAULT_TABLE table=default.Table_8e48ce56b9fb4e15bfc3b49e8273b872 [id=95d0ac821b504fe59dab30ccceb1b96f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.741310 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_9c23aecb8f9240049865c0e9db0f25b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:46.741722 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81c56c2cd7fa45f390602adb67681f04. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:46.742111 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c23aecb8f9240049865c0e9db0f25b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.744145 3282 tablet_copy_client.cc:323] T 164f76edcf8f4bd498dcd2c25fd7477d P f1e775296eb746db8b318f17d26820e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.104.2:44451
I20250112 02:52:46.744349 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_8c1228cf5d9a4355811e6430527d39b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.744838 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c1228cf5d9a4355811e6430527d39b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.745332 899 tablet_service.cc:1467] Processing CreateTablet for tablet 1d909275d67c4127b3144de53efc7d3e (DEFAULT_TABLE table=default.Table_ade612ff606d477c84a3de161a684cee [id=97365c7c50c94b049acbe232693c3543]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.745770 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d909275d67c4127b3144de53efc7d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.752394 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_9f558b53671c40849c3fe1c50dba42d8" } new_table_name: "default.Table_67b6f55a9ba84491bdda41cdf82c25fc" modify_external_catalogs: true
I20250112 02:52:46.756564 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 432) took 1881 ms. Trace:
I20250112 02:52:46.757869 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_479d0be56c26459c938d6fb9c52d4644"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.758389 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_479d0be56c26459c938d6fb9c52d4644 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.760111 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_2fb335949dc24790a343c60f455a560a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.760673 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fb335949dc24790a343c60f455a560a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.765007 895 tablet_service.cc:1467] Processing CreateTablet for tablet 81c56c2cd7fa45f390602adb67681f04 (DEFAULT_TABLE table=default.Table_8e48ce56b9fb4e15bfc3b49e8273b872 [id=95d0ac821b504fe59dab30ccceb1b96f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.765478 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81c56c2cd7fa45f390602adb67681f04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.770942 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 38e4a69f470a4d62b865e2dd594fab63 (DEFAULT_TABLE table=default.Table_c7dcf1defec042dabf093c4b9b0449a7 [id=77b9615977a74bceb6f208fb5a28742f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.771274 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38e4a69f470a4d62b865e2dd594fab63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.774798 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c12a1c5e3584450ac193743a0107e80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:46.775269 3138 raft_consensus.cc:513] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.775290 3109 rpcz_store.cc:276] 0112 02:52:44.830818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.835032 (+ 4214us) service_pool.cc:225] Handling call
0112 02:52:46.708100 (+1873068us) ts_tablet_manager.cc:1145] Deleting tablet 76009bac22ce4136b148b493a004a0f7
0112 02:52:46.708113 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.712530 (+ 4417us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1872921}
I20250112 02:52:46.778261 3111 ts_tablet_manager.cc:1918] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:46.778375 3111 log.cc:1198] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/61eac6b572ff47c185768bee7490edf7
I20250112 02:52:46.778741 3111 ts_tablet_manager.cc:1939] T 61eac6b572ff47c185768bee7490edf7 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:46.783334 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 433) took 1410 ms. Trace:
I20250112 02:52:46.783897 3111 rpcz_store.cc:276] 0112 02:52:45.373265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.424900 (+ 51635us) service_pool.cc:225] Handling call
0112 02:52:46.670642 (+1245742us) ts_tablet_manager.cc:1145] Deleting tablet 61eac6b572ff47c185768bee7490edf7
0112 02:52:46.670650 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.732830 (+ 62180us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.778176 (+ 45346us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.783312 (+ 5136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1245448}
I20250112 02:52:46.784649 3111 tablet_replica.cc:331] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:46.785089 3111 raft_consensus.cc:2238] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:46.785728 3111 raft_consensus.cc:2267] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.786720 3111 ts_tablet_manager.cc:1905] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.788273 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 17c54c5dc86446cf85974777c8145ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:46.788349 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 13578542e9e741e8817e0ee63b10f1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:35 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:46.788841 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 99407ead9cf547e3bbe78c0d0b327e6a with delete_type TABLET_DATA_DELETED (replaced by tablet a94cec0f0d4d4f06aa5667135efbf5a6) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:46.787792 3266 raft_consensus.cc:491] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:46.789646 3266 raft_consensus.cc:513] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:46.790549 713 leader_election.cc:343] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: c29a9ca43e034b43a00207e5847bbee3
I20250112 02:52:46.790617 749 tablet_service.cc:1514] Processing DeleteTablet for tablet c29a9ca43e034b43a00207e5847bbee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:46.790838 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe57d374eb44c3fb1532484e214b654 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:46.790442 3266 leader_election.cc:290] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:46.791612 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d3664071b146a3ac7c6885f9bda789" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:46.791849 914 raft_consensus.cc:2463] T 54d3664071b146a3ac7c6885f9bda789 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:46.792232 989 leader_election.cc:304] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:46.792448 3266 raft_consensus.cc:2798] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:46.792573 3266 raft_consensus.cc:491] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:46.792712 3266 raft_consensus.cc:3054] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:46.792871 1028 tablet_service.cc:1467] Processing CreateTablet for tablet be44eccc90d24b5ea5f8fcfc8a74e039 (DEFAULT_TABLE table=default.Table_a12ca3f6249040c8bc457ee894d7a274 [id=3f439ae9d2ed4b009661a02117df21c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.793210 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be44eccc90d24b5ea5f8fcfc8a74e039. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.795145 3266 raft_consensus.cc:513] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.795841 3245 tablet_copy_client.cc:323] T ca6f080e5fd84084b601b646d4e6c9e2 P f1e775296eb746db8b318f17d26820e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.104.2:44451
I20250112 02:52:46.796132 3266 leader_election.cc:290] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:46.806491 3111 ts_tablet_manager.cc:1918] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:46.807665 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_f8a40c27bebc483cbf62d5d0a4c7fb24" } new_table_name: "default.Table_ef71f75b40b14fa5b96ee12db588b5d8" modify_external_catalogs: true
I20250112 02:52:46.807694 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_b47e051ef51c499dbda95bbe0b3cea50" } new_table_name: "default.Table_b93a3e5a4b5c4739bc3153d38c77fabd" modify_external_catalogs: true
I20250112 02:52:46.816532 3260 raft_consensus.cc:491] T d86417fc1c844cdf85323b94c6dce6d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:46.816726 3260 raft_consensus.cc:513] T d86417fc1c844cdf85323b94c6dce6d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:46.817375 3260 leader_election.cc:290] T d86417fc1c844cdf85323b94c6dce6d5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:46.817500 3111 log.cc:1198] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/e4d42f1611de49d6b6e5aaa9be907054
I20250112 02:52:46.818084 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86417fc1c844cdf85323b94c6dce6d5" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:46.817963 3111 ts_tablet_manager.cc:1939] T e4d42f1611de49d6b6e5aaa9be907054 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:46.819243 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 435) took 1196 ms. Trace:
I20250112 02:52:46.819667 3111 rpcz_store.cc:276] 0112 02:52:45.623004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.623060 (+ 56us) service_pool.cc:225] Handling call
0112 02:52:46.784602 (+1161542us) ts_tablet_manager.cc:1145] Deleting tablet e4d42f1611de49d6b6e5aaa9be907054
0112 02:52:46.784618 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.803780 (+ 19162us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.806419 (+ 2639us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.819230 (+ 12811us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1161255}
I20250112 02:52:46.820461 3111 tablet_replica.cc:331] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
W20250112 02:52:46.820827 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet efa6cc11872f4e67a9e7b73bd65651fa consensus state has no leader
I20250112 02:52:46.820832 3111 raft_consensus.cc:2238] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:46.821386 3111 raft_consensus.cc:2267] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.821632 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_4194fa71a3a8428799fb963cd3282e24" } modify_external_catalogs: true
I20250112 02:52:46.822492 3111 ts_tablet_manager.cc:1905] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.822892 452 catalog_manager.cc:5526] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:46.823940 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_4194fa71a3a8428799fb963cd3282e24" } modify_external_catalogs: true
I20250112 02:52:46.825258 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 442) took 1903 ms. Trace:
I20250112 02:52:46.825399 3109 rpcz_store.cc:276] 0112 02:52:44.874278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.874493 (+ 215us) service_pool.cc:225] Handling call
0112 02:52:46.775827 (+1901334us) ts_tablet_manager.cc:1145] Deleting tablet f3771166b77e4f90aaa116ab1a9dc299
0112 02:52:46.775842 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.777427 (+ 1585us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1901150}
I20250112 02:52:46.825884 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 443) took 1923 ms. Trace:
I20250112 02:52:46.825968 3109 rpcz_store.cc:276] 0112 02:52:44.902080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.907514 (+ 5434us) service_pool.cc:225] Handling call
0112 02:52:46.825715 (+1918201us) ts_tablet_manager.cc:1145] Deleting tablet 940affdc2c484b7d9fd2de17f9d1cea7
0112 02:52:46.825730 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.825850 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1917996}
I20250112 02:52:46.826324 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 444) took 1917 ms. Trace:
I20250112 02:52:46.826402 3109 rpcz_store.cc:276] 0112 02:52:44.909152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.909205 (+ 53us) service_pool.cc:225] Handling call
0112 02:52:46.826176 (+1916971us) ts_tablet_manager.cc:1145] Deleting tablet bab4978edea0445fb0456e88e7fb4c44
0112 02:52:46.826202 (+ 26us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.826292 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1916796}
I20250112 02:52:46.826874 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet a470f84d50df4cf0b4a1e5d248b4c32a (table default.Table_97a56432deb04f0ea0d6f838de79af39 [id=1fa1474ab8804667b22e5f30c48c1630]). Expected version 1 got 0
I20250112 02:52:46.827453 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:46.827998 749 tablet_service.cc:1467] Processing CreateTablet for tablet 81c56c2cd7fa45f390602adb67681f04 (DEFAULT_TABLE table=default.Table_8e48ce56b9fb4e15bfc3b49e8273b872 [id=95d0ac821b504fe59dab30ccceb1b96f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.828217 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_5e1a058646834ea99def3d8262bee81b" } modify_external_catalogs: true
I20250112 02:52:46.828392 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81c56c2cd7fa45f390602adb67681f04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.828361 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_5e1a058646834ea99def3d8262bee81b" } modify_external_catalogs: true
I20250112 02:52:46.827848 452 catalog_manager.cc:5526] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:46.829136 452 catalog_manager.cc:5526] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:46.830026 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:46.830067 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_7ee8a748c60d439088423dcb284b17ad" } new_table_name: "default.Table_c0a21e51b1284e2ca2d529c6f24c81fa" modify_external_catalogs: true
W20250112 02:52:46.831077 854 leader_election.cc:343] T d86417fc1c844cdf85323b94c6dce6d5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:46.831521 3106 consensus_queue.cc:1035] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20250112 02:52:46.833164 3138 leader_election.cc:290] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:46.834331 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d3664071b146a3ac7c6885f9bda789" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:46.836438 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_4e1bfc47f727446ba5be2309a37017fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.836905 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e1bfc47f727446ba5be2309a37017fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:46.839185 991 leader_election.cc:343] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:46.839444 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:39987, user_credentials={real_user=slave}} blocked reactor thread for 56996.1us
I20250112 02:52:46.840005 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 61eac6b572ff47c185768bee7490edf7 (table default.Table_e8e62bd2fffe4f50849f722dbece2a2c [id=3603f999c77a48dca737bc8ff74eb9de]) successfully deleted
I20250112 02:52:46.840292 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet e4d42f1611de49d6b6e5aaa9be907054 (table default.Table_f952a0479e8d478bbd9484fc5bf041c8 [id=c01b653cd9fe45b2af721eb97936c5d4]) successfully deleted
I20250112 02:52:46.841432 898 tablet_service.cc:1467] Processing CreateTablet for tablet 38e4a69f470a4d62b865e2dd594fab63 (DEFAULT_TABLE table=default.Table_c7dcf1defec042dabf093c4b9b0449a7 [id=77b9615977a74bceb6f208fb5a28742f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.842150 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38e4a69f470a4d62b865e2dd594fab63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.843339 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 1784b6e1337a440ebbae8eb885a51fca (DEFAULT_TABLE table=default.Table_c8d6f1dc529c44efa678ba41a8cde7e9 [id=376101d8628a4ffd9318fdc5f34b6c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.843433 887 tablet_service.cc:1467] Processing CreateTablet for tablet 1784b6e1337a440ebbae8eb885a51fca (DEFAULT_TABLE table=default.Table_c8d6f1dc529c44efa678ba41a8cde7e9 [id=376101d8628a4ffd9318fdc5f34b6c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.843847 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1784b6e1337a440ebbae8eb885a51fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.844853 890 tablet_service.cc:1467] Processing CreateTablet for tablet 51334a07328c4e7d92c843dd18907671 (DEFAULT_TABLE table=default.Table_f37babaaa3a241778a5a5542b9b832b0 [id=fcad044651084d92b3b57aa272cbedc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.845065 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet b768b7534e034bff86045e8a474bf05d
I20250112 02:52:46.845163 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51334a07328c4e7d92c843dd18907671. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.845887 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b768b7534e034bff86045e8a474bf05d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.849663 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_ee16ba338f554f4dbd75b332adf897f3" } new_table_name: "default.Table_bb0a4f973f234c80954ac449b0c6e88a" modify_external_catalogs: true
I20250112 02:52:46.849872 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_a5b8c2cd15dd47818378fc3a3065b890"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.850323 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5b8c2cd15dd47818378fc3a3065b890 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.776168 939 tablet_copy_service.cc:140] P fe36951a83fb4e57b26e187edbcbfe3d: Received BeginTabletCopySession request for tablet 164f76edcf8f4bd498dcd2c25fd7477d from peer f1e775296eb746db8b318f17d26820e6 ({username='slave'} at 127.0.104.3:52739)
I20250112 02:52:46.855789 939 tablet_copy_service.cc:161] P fe36951a83fb4e57b26e187edbcbfe3d: Beginning new tablet copy session on tablet 164f76edcf8f4bd498dcd2c25fd7477d from peer f1e775296eb746db8b318f17d26820e6 at {username='slave'} at 127.0.104.3:52739: session id = f1e775296eb746db8b318f17d26820e6-164f76edcf8f4bd498dcd2c25fd7477d
I20250112 02:52:46.796456 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d3664071b146a3ac7c6885f9bda789" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:46.855908 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_1950cb8738a5407a84000b99c2cf5e2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:46.856027 914 raft_consensus.cc:3054] T 54d3664071b146a3ac7c6885f9bda789 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:46.856389 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1950cb8738a5407a84000b99c2cf5e2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.856209 938 tablet_copy_service.cc:140] P fe36951a83fb4e57b26e187edbcbfe3d: Received BeginTabletCopySession request for tablet ca6f080e5fd84084b601b646d4e6c9e2 from peer f1e775296eb746db8b318f17d26820e6 ({username='slave'} at 127.0.104.3:52739)
I20250112 02:52:46.861574 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-tasko6SHYo/build/asan/bin/kudu
/tmp/dist-test-tasko6SHYo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:39987
--webserver_interface=127.0.104.60
--webserver_port=42831
--builtin_ntp_servers=127.0.104.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42341,127.0.104.61:38771,127.0.104.60:39987
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 02:52:46.856590 899 tablet_service.cc:1467] Processing CreateTablet for tablet be44eccc90d24b5ea5f8fcfc8a74e039 (DEFAULT_TABLE table=default.Table_a12ca3f6249040c8bc457ee894d7a274 [id=3f439ae9d2ed4b009661a02117df21c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.862273 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be44eccc90d24b5ea5f8fcfc8a74e039. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.862583 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1784b6e1337a440ebbae8eb885a51fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.863619 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 09900b42e8a443e680cc35c96a1dbbb7
I20250112 02:52:46.865603 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 09900b42e8a443e680cc35c96a1dbbb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.869235 1035 tablet_service.cc:1467] Processing CreateTablet for tablet a1ea1a1491a945049f9f4164eea7535c (DEFAULT_TABLE table=default.Table_83ec35a32e3e47fc9ccdde96b4f221df [id=ae920b41954f4968b81659c0b2a8e2cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.869570 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ea1a1491a945049f9f4164eea7535c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.869849 938 tablet_copy_service.cc:161] P fe36951a83fb4e57b26e187edbcbfe3d: Beginning new tablet copy session on tablet ca6f080e5fd84084b601b646d4e6c9e2 from peer f1e775296eb746db8b318f17d26820e6 at {username='slave'} at 127.0.104.3:52739: session id = f1e775296eb746db8b318f17d26820e6-ca6f080e5fd84084b601b646d4e6c9e2
I20250112 02:52:46.874150 890 tablet_service.cc:1467] Processing CreateTablet for tablet a1ea1a1491a945049f9f4164eea7535c (DEFAULT_TABLE table=default.Table_83ec35a32e3e47fc9ccdde96b4f221df [id=ae920b41954f4968b81659c0b2a8e2cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.874533 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ea1a1491a945049f9f4164eea7535c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.880331 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_0c453685abe143aa82c64b441ec582bb [id=097d902a92934882bc7c016da518642a] alter complete (version 3)
I20250112 02:52:46.881806 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 61eac6b572ff47c185768bee7490edf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.881961 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 781a9943f705477cb6eae16df4468d2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:41 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.882404 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet f3771166b77e4f90aaa116ab1a9dc299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:39 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.844070 3111 ts_tablet_manager.cc:1918] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 02:52:46.852286 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 51334a07328c4e7d92c843dd18907671 (DEFAULT_TABLE table=default.Table_f37babaaa3a241778a5a5542b9b832b0 [id=fcad044651084d92b3b57aa272cbedc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.883281 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51334a07328c4e7d92c843dd18907671. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.887276 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 26e2baf248444134a391242bad358ebe (DEFAULT_TABLE table=default.Table_468eef2886374c71b52269b9a47a4772 [id=0a2e5e706d9a4ca89f7620c3808dec66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.852708 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet b768b7534e034bff86045e8a474bf05d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.882931 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e7882e07784bdda18dc1ffd20963c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.887735 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26e2baf248444134a391242bad358ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.888540 891 tablet_service.cc:1467] Processing CreateTablet for tablet 1bd1fb8d69bd411bb02fdafa6ba720f5 (DEFAULT_TABLE table=default.Table_6a2dccba50e94d42b01b4ad9e1ca5a35 [id=e8fe98863848407f97eb0c8c4d3c8235]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.888895 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 1b34cef7c8a14dbda7c091dfed46515a (DEFAULT_TABLE table=default.Table_ea6f20bc46ec487cae9e3abf6419c80c [id=34602f2c95734d478e38e1847de46aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.888944 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bd1fb8d69bd411bb02fdafa6ba720f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.889164 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b34cef7c8a14dbda7c091dfed46515a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.891695 889 tablet_service.cc:1467] Processing CreateTablet for tablet 26e2baf248444134a391242bad358ebe (DEFAULT_TABLE table=default.Table_468eef2886374c71b52269b9a47a4772 [id=0a2e5e706d9a4ca89f7620c3808dec66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.891974 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26e2baf248444134a391242bad358ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.895242 1028 tablet_service.cc:1467] Processing CreateTablet for tablet b3392229868e4b08a16874f4a1b76de2 (DEFAULT_TABLE table=default.Table_7ca45e521bed4ddf8d26c9d32651eb34 [id=3f152d4fbcf3491e911f1a6716370f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.895567 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3392229868e4b08a16874f4a1b76de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.897374 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 1bd1fb8d69bd411bb02fdafa6ba720f5 (DEFAULT_TABLE table=default.Table_6a2dccba50e94d42b01b4ad9e1ca5a35 [id=e8fe98863848407f97eb0c8c4d3c8235]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.897930 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bd1fb8d69bd411bb02fdafa6ba720f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.900208 1017 tablet_service.cc:1467] Processing CreateTablet for tablet bbaa7dfc7da14cc2b4f3731d3da6107b (DEFAULT_TABLE table=default.Table_fb9557bec89748dabf949080f016deba [id=e0e08894993647c69bc6a75ec9f4bcbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.900564 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbaa7dfc7da14cc2b4f3731d3da6107b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.901974 1023 tablet_service.cc:1467] Processing CreateTablet for tablet ee0c474eedcd459697261747e802279e (DEFAULT_TABLE table=default.Table_65dfa36d9f664916bf107d10764f88cd [id=39a1a01693ae4665b769712b85ba5311]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.902271 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee0c474eedcd459697261747e802279e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.905560 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_23c987a24fed4ceeb90fd25f0367b08f" } modify_external_catalogs: true
I20250112 02:52:46.905782 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_23c987a24fed4ceeb90fd25f0367b08f" } modify_external_catalogs: true
I20250112 02:52:46.906611 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_1a098348f5684cebac211f561607c408" } modify_external_catalogs: true
I20250112 02:52:46.906761 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_1a098348f5684cebac211f561607c408" } modify_external_catalogs: true
I20250112 02:52:46.907828 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 2473488c4890403291e38f287a976851 (DEFAULT_TABLE table=default.Table_7f1cd8fa7b844d0fba0cad8d1a6652e6 [id=cc8ce7ac6e4a416486ad1175ca1e1d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.908082 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2473488c4890403291e38f287a976851. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.910511 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_dc292752faf041ed8ca40958c86765b0" } modify_external_catalogs: true
I20250112 02:52:46.910645 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_dc292752faf041ed8ca40958c86765b0" } modify_external_catalogs: true
I20250112 02:52:46.911739 897 tablet_service.cc:1467] Processing CreateTablet for tablet b3392229868e4b08a16874f4a1b76de2 (DEFAULT_TABLE table=default.Table_7ca45e521bed4ddf8d26c9d32651eb34 [id=3f152d4fbcf3491e911f1a6716370f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.912096 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3392229868e4b08a16874f4a1b76de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.915066 893 tablet_service.cc:1467] Processing CreateTablet for tablet bbaa7dfc7da14cc2b4f3731d3da6107b (DEFAULT_TABLE table=default.Table_fb9557bec89748dabf949080f016deba [id=e0e08894993647c69bc6a75ec9f4bcbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.915357 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbaa7dfc7da14cc2b4f3731d3da6107b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.919896 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 648960d4f08f4ce8a4f00aeeb34048f5
I20250112 02:52:46.921574 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet a7db121c542f423bbffd2e36c459c3f0
I20250112 02:52:46.922670 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 648960d4f08f4ce8a4f00aeeb34048f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.923310 892 tablet_service.cc:1467] Processing CreateTablet for tablet 2473488c4890403291e38f287a976851 (DEFAULT_TABLE table=default.Table_7f1cd8fa7b844d0fba0cad8d1a6652e6 [id=cc8ce7ac6e4a416486ad1175ca1e1d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.923751 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2473488c4890403291e38f287a976851. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.925127 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet a7db121c542f423bbffd2e36c459c3f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.925135 914 raft_consensus.cc:2463] T 54d3664071b146a3ac7c6885f9bda789 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:46.925376 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 648960d4f08f4ce8a4f00aeeb34048f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.927681 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 2796289c7e8f4b45b26508c50b57c640
I20250112 02:52:46.928264 1029 tablet_service.cc:1467] Processing CreateTablet for tablet f01da4bb769843e5b81c0c6ba0ab9db2 (DEFAULT_TABLE table=default.Table_bfcd8a99635e43a29c4a91082cee18f4 [id=0332251275a74598ba6ee151af0481c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.928361 894 tablet_service.cc:1514] Processing DeleteTablet for tablet a7db121c542f423bbffd2e36c459c3f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.928792 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f01da4bb769843e5b81c0c6ba0ab9db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.930728 894 tablet_service.cc:1467] Processing CreateTablet for tablet ee0c474eedcd459697261747e802279e (DEFAULT_TABLE table=default.Table_65dfa36d9f664916bf107d10764f88cd [id=39a1a01693ae4665b769712b85ba5311]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.928514 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 2796289c7e8f4b45b26508c50b57c640 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:46.928293 989 leader_election.cc:304] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:46.931356 2977 raft_consensus.cc:2798] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:46.931499 2977 raft_consensus.cc:695] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:46.931635 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee0c474eedcd459697261747e802279e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.931727 2977 consensus_queue.cc:237] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.932868 897 tablet_service.cc:1467] Processing CreateTablet for tablet f01da4bb769843e5b81c0c6ba0ab9db2 (DEFAULT_TABLE table=default.Table_bfcd8a99635e43a29c4a91082cee18f4 [id=0332251275a74598ba6ee151af0481c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.933219 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f01da4bb769843e5b81c0c6ba0ab9db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.936496 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 7d0d84297810420b8b7a4b24841eda55 (DEFAULT_TABLE table=default.Table_35eb917609ca4728bf4823fb912a9680 [id=80a612296e3742ee8906307445cc2ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.936607 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea806f38475444c8f3d67e32bec9244 (DEFAULT_TABLE table=default.Table_43f7ba33caf249c4ba386a9199462957 [id=6a6c608131944f858e911d7f376f2813]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.937386 1026 tablet_service.cc:1467] Processing CreateTablet for tablet b5485bdf54ab40b8b2b9b0bbac522468 (DEFAULT_TABLE table=default.Table_ef4fac0bfa544b4da7abc353a32d4e6b [id=56b2f9b180e744c4a079330bae08ac84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.938097 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_a6857eb18f404b15a97dade22ad383c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.938649 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6857eb18f404b15a97dade22ad383c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.939597 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_e968295bfd5546dfb74184d506b542e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.940063 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e968295bfd5546dfb74184d506b542e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:46.941066 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet efa6cc11872f4e67a9e7b73bd65651fa (table default.Table_b93a3e5a4b5c4739bc3153d38c77fabd [id=be97fe5bb0e7414f8efdffdf24465295], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet efa6cc11872f4e67a9e7b73bd65651fa consensus state has no leader
I20250112 02:52:46.944487 3111 log.cc:1198] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/09d06e2e8c13464db8cf801a292bdc99
I20250112 02:52:46.945080 3111 ts_tablet_manager.cc:1939] T 09d06e2e8c13464db8cf801a292bdc99 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:46.951471 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d0d84297810420b8b7a4b24841eda55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.952978 894 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea806f38475444c8f3d67e32bec9244 (DEFAULT_TABLE table=default.Table_43f7ba33caf249c4ba386a9199462957 [id=6a6c608131944f858e911d7f376f2813]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.953284 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 2796289c7e8f4b45b26508c50b57c640 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:46.955128 887 tablet_service.cc:1467] Processing CreateTablet for tablet b5485bdf54ab40b8b2b9b0bbac522468 (DEFAULT_TABLE table=default.Table_ef4fac0bfa544b4da7abc353a32d4e6b [id=56b2f9b180e744c4a079330bae08ac84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.958663 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 2329b3ddfae44b96a637ed00247f919d (DEFAULT_TABLE table=default.Table_84d9fd4fb68b4c31a7f438c12f67d042 [id=938d0e6585be408e8a842d9b9964c48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.958683 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_b910dc09eda7442ea8f6df224e60e7db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:46.958994 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2329b3ddfae44b96a637ed00247f919d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:46.959193 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b910dc09eda7442ea8f6df224e60e7db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.961139 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_a1e87ce18198488dac5e9cbbb10fd42e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.961598 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1e87ce18198488dac5e9cbbb10fd42e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.963371 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 436) took 1284 ms. Trace:
I20250112 02:52:46.963589 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5485bdf54ab40b8b2b9b0bbac522468. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.964578 3111 rpcz_store.cc:276] 0112 02:52:45.678950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.679027 (+ 77us) service_pool.cc:225] Handling call
0112 02:52:46.820419 (+1141392us) ts_tablet_manager.cc:1145] Deleting tablet 09d06e2e8c13464db8cf801a292bdc99
0112 02:52:46.820427 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.831408 (+ 10981us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.843987 (+ 12579us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.963356 (+119369us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1141202}
I20250112 02:52:46.965004 3111 tablet_replica.cc:331] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:46.965474 3111 raft_consensus.cc:2238] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:46.966028 3111 raft_consensus.cc:2267] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:46.955652 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ea806f38475444c8f3d67e32bec9244. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.966173 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_7921b99e24f84cceb58128f18925ad49" } new_table_name: "default.Table_f2431b9445b84d008a9902efe7bbcd66" modify_external_catalogs: true
I20250112 02:52:46.966601 3111 ts_tablet_manager.cc:1905] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:46.967792 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ea806f38475444c8f3d67e32bec9244. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.957470 939 tablet_copy_source_session.cc:215] T 164f76edcf8f4bd498dcd2c25fd7477d P fe36951a83fb4e57b26e187edbcbfe3d: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 02:52:46.969604 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_1ccecb4c9e1f481b9c467fea54894390"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:46.970333 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5485bdf54ab40b8b2b9b0bbac522468. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.972776 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 09d06e2e8c13464db8cf801a292bdc99 (table default.Table_2c52fead633649dd869763e5aa532be7 [id=e3bb489c53454222a79702b798c0c2f1]) successfully deleted
I20250112 02:52:46.972957 3282 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 164f76edcf8f4bd498dcd2c25fd7477d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.974208 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_b7da5ba607bd47e89d807929a64e2ea7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.974622 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7da5ba607bd47e89d807929a64e2ea7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.975788 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_75a41a4f8be84a56b1103d08e0510d4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:46.976186 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75a41a4f8be84a56b1103d08e0510d4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:46.976297 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ccecb4c9e1f481b9c467fea54894390 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:46.976653 2745 raft_consensus.cc:491] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:46.976776 2745 raft_consensus.cc:513] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:46.977309 2745 leader_election.cc:290] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:46.980260 892 tablet_service.cc:1467] Processing CreateTablet for tablet 2329b3ddfae44b96a637ed00247f919d (DEFAULT_TABLE table=default.Table_84d9fd4fb68b4c31a7f438c12f67d042 [id=938d0e6585be408e8a842d9b9964c48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.980587 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2329b3ddfae44b96a637ed00247f919d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.981808 1028 tablet_service.cc:1467] Processing CreateTablet for tablet efa6cc11872f4e67a9e7b73bd65651fa (DEFAULT_TABLE table=default.Table_b93a3e5a4b5c4739bc3153d38c77fabd [id=be97fe5bb0e7414f8efdffdf24465295]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.982161 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efa6cc11872f4e67a9e7b73bd65651fa. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:46.983213 989 leader_election.cc:343] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: f3771166b77e4f90aaa116ab1a9dc299
W20250112 02:52:46.983897 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f559dedfb7854cf890bef362c50b4213 consensus state has no leader
I20250112 02:52:46.987876 893 tablet_service.cc:1467] Processing CreateTablet for tablet efa6cc11872f4e67a9e7b73bd65651fa (DEFAULT_TABLE table=default.Table_b93a3e5a4b5c4739bc3153d38c77fabd [id=be97fe5bb0e7414f8efdffdf24465295]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.988195 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efa6cc11872f4e67a9e7b73bd65651fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.990581 3282 tablet_copy_client.cc:806] T 164f76edcf8f4bd498dcd2c25fd7477d P f1e775296eb746db8b318f17d26820e6: tablet copy: Starting download of 0 data blocks...
I20250112 02:52:46.990816 3282 tablet_copy_client.cc:670] T 164f76edcf8f4bd498dcd2c25fd7477d P f1e775296eb746db8b318f17d26820e6: tablet copy: Starting download of 1 WAL segments...
I20250112 02:52:46.992061 880 tablet_service.cc:1467] Processing CreateTablet for tablet b1cb48c936844d049a1c832b47986d73 (DEFAULT_TABLE table=default.Table_5aeb2d7bb68c46a589a48526dcc40aaf [id=9d267774bf59428e854651956fbd79be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.992388 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1cb48c936844d049a1c832b47986d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.992476 891 tablet_service.cc:1467] Processing CreateTablet for tablet 1b34cef7c8a14dbda7c091dfed46515a (DEFAULT_TABLE table=default.Table_ea6f20bc46ec487cae9e3abf6419c80c [id=34602f2c95734d478e38e1847de46aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.992734 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b34cef7c8a14dbda7c091dfed46515a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.995729 886 tablet_service.cc:1467] Processing CreateTablet for tablet 5b78e2f2303348dea1ca11c1cfd24e3f (DEFAULT_TABLE table=default.Table_8bb2bb131d6b4262b964926902af01d3 [id=8128f3bba480478993bc9b24e1d16c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:46.996043 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b78e2f2303348dea1ca11c1cfd24e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.002943 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 777272e7159b438ebff4db93698411d0 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:46.992084 889 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca98a3f0fac4d2590c2c88aea34281e (DEFAULT_TABLE table=default.Table_7077f9a718664d8b915ee4003502306a [id=6df53b1d0c7a4fa1a18b3202741630b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.009238 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca98a3f0fac4d2590c2c88aea34281e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.009487 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 777272e7159b438ebff4db93698411d0
I20250112 02:52:47.010286 447 catalog_manager.cc:6596] ReplaceTablet: tablet 777272e7159b438ebff4db93698411d0 deleted and replaced by tablet 0925ba2197bb461190b832d062567040
I20250112 02:52:47.010632 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 777272e7159b438ebff4db93698411d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 0925ba2197bb461190b832d062567040) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:47.010939 3282 tablet_copy_client.cc:538] T 164f76edcf8f4bd498dcd2c25fd7477d P f1e775296eb746db8b318f17d26820e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 02:52:47.016234 3273 raft_consensus.cc:491] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:47.016438 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 777272e7159b438ebff4db93698411d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 0925ba2197bb461190b832d062567040) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:47.019824 1017 tablet_service.cc:1467] Processing CreateTablet for tablet b1cb48c936844d049a1c832b47986d73 (DEFAULT_TABLE table=default.Table_5aeb2d7bb68c46a589a48526dcc40aaf [id=9d267774bf59428e854651956fbd79be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.020274 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1cb48c936844d049a1c832b47986d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.016363 3273 raft_consensus.cc:513] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:47.020732 447 master_service.cc:946] ReplaceTablet: received request to replace tablet b768b7534e034bff86045e8a474bf05d from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:47.021768 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c54c5dc86446cf85974777c8145ef6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:47.022228 854 leader_election.cc:343] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:47.022475 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_096703f878244b8dac2dcf187fd23fd6" } new_table_name: "default.Table_36ee8cc4670048c6b9c96ac874785d48" modify_external_catalogs: true
I20250112 02:52:47.023033 1022 tablet_service.cc:1467] Processing CreateTablet for tablet a7db121c542f423bbffd2e36c459c3f0 (DEFAULT_TABLE table=default.Table_dc292752faf041ed8ca40958c86765b0 [id=407e1e8894524b7682f498b3904b06e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.025745 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_f69a79e767cb4d6da20e31ddb7418a63 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 1 got 0
I20250112 02:52:47.027611 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7db121c542f423bbffd2e36c459c3f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.031569 3273 leader_election.cc:290] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:47.034627 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 5b78e2f2303348dea1ca11c1cfd24e3f (DEFAULT_TABLE table=default.Table_8bb2bb131d6b4262b964926902af01d3 [id=8128f3bba480478993bc9b24e1d16c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.034991 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b78e2f2303348dea1ca11c1cfd24e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.038008 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca98a3f0fac4d2590c2c88aea34281e (DEFAULT_TABLE table=default.Table_7077f9a718664d8b915ee4003502306a [id=6df53b1d0c7a4fa1a18b3202741630b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.038442 1181 tablet_bootstrap.cc:492] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:47.038573 1181 ts_tablet_manager.cc:1397] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.729s user 0.006s sys 0.002s
I20250112 02:52:47.038864 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_2add53a7889440009ab2d3b0e58c5381" } new_table_name: "default.Table_5894b4f21573414ca72f6155c0862ccf" modify_external_catalogs: true
I20250112 02:52:47.042351 886 tablet_service.cc:1467] Processing CreateTablet for tablet a7db121c542f423bbffd2e36c459c3f0 (DEFAULT_TABLE table=default.Table_dc292752faf041ed8ca40958c86765b0 [id=407e1e8894524b7682f498b3904b06e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.042671 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7db121c542f423bbffd2e36c459c3f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:46.957777 897 tablet_service.cc:1467] Processing CreateTablet for tablet 7d0d84297810420b8b7a4b24841eda55 (DEFAULT_TABLE table=default.Table_35eb917609ca4728bf4823fb912a9680 [id=80a612296e3742ee8906307445cc2ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.050482 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d0d84297810420b8b7a4b24841eda55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.052488 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_f38c3f9c44b741219f03cf46cfbc9937" } modify_external_catalogs: true
I20250112 02:52:47.052668 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_f38c3f9c44b741219f03cf46cfbc9937" } modify_external_catalogs: true
I20250112 02:52:47.055877 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 9fae49da375f4c6a894c1237dd6621fb from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:47.059087 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_18c158c848f1430f9852655c57f10933" } new_table_name: "default.Table_9ecbd17bdc664accbaf672ff5590eeca" modify_external_catalogs: true
I20250112 02:52:47.060772 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86417fc1c844cdf85323b94c6dce6d5" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:47.062595 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_65742cc1c2824757830aec593a96efd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.063059 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65742cc1c2824757830aec593a96efd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.063439 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_c1820d9fe8904d029ea65d852fcea59b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.064013 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1820d9fe8904d029ea65d852fcea59b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.066272 3282 tablet_bootstrap.cc:492] T 164f76edcf8f4bd498dcd2c25fd7477d P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:47.066500 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d3664071b146a3ac7c6885f9bda789" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:47.070075 760 tablet_service.cc:1467] Processing CreateTablet for tablet 1d909275d67c4127b3144de53efc7d3e (DEFAULT_TABLE table=default.Table_ade612ff606d477c84a3de161a684cee [id=97365c7c50c94b049acbe232693c3543]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.070186 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 9fae49da375f4c6a894c1237dd6621fb
I20250112 02:52:47.070638 446 catalog_manager.cc:6596] ReplaceTablet: tablet 9fae49da375f4c6a894c1237dd6621fb deleted and replaced by tablet f203c86211f94999951740e1e6351b19
I20250112 02:52:46.857908 3110 ts_tablet_manager.cc:1918] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:47.072160 3110 log.cc:1198] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/d95f98ae7bdf42518bbff55e257fea88
I20250112 02:52:47.072624 3110 ts_tablet_manager.cc:1939] T d95f98ae7bdf42518bbff55e257fea88 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:47.073596 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 354) took 1891 ms. Trace:
I20250112 02:52:47.073824 3110 rpcz_store.cc:276] 0112 02:52:45.181744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.182287 (+ 543us) service_pool.cc:225] Handling call
0112 02:52:46.577112 (+1394825us) ts_tablet_manager.cc:1145] Deleting tablet d95f98ae7bdf42518bbff55e257fea88
0112 02:52:46.577120 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:46.581242 (+ 4122us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:46.857833 (+276591us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:47.073575 (+215742us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1394713}
I20250112 02:52:47.073997 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_c7107062f9574066bf9a4a362f9b3cfb" } new_table_name: "default.Table_4ce8a06de05a4423a9744ed9fb6c2ec4" modify_external_catalogs: true
I20250112 02:52:47.074447 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet d95f98ae7bdf42518bbff55e257fea88 (table default.Table_2fcdbb05970f4a098f7ba3e287fa0348 [id=70c61753e9944e1fbd65e9ec41a48ee2]) successfully deleted
I20250112 02:52:47.074563 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 355) took 1877 ms. Trace:
I20250112 02:52:47.074630 3110 rpcz_store.cc:276] 0112 02:52:45.197396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.197438 (+ 42us) service_pool.cc:225] Handling call
0112 02:52:47.074456 (+1877018us) ts_tablet_manager.cc:1145] Deleting tablet 64b0d58af4cf4adcbeb3ef5070f2982d
0112 02:52:47.074464 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.074549 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1876863}
I20250112 02:52:47.075029 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 356) took 1847 ms. Trace:
I20250112 02:52:47.075101 3110 rpcz_store.cc:276] 0112 02:52:45.227363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.227445 (+ 82us) service_pool.cc:225] Handling call
0112 02:52:47.074943 (+1847498us) ts_tablet_manager.cc:1145] Deleting tablet be856e4b913e40d58a3f3ba54698cc5f
0112 02:52:47.074950 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.075016 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1847330}
W20250112 02:52:47.075387 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet be856e4b913e40d58a3f3ba54698cc5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be856e4b913e40d58a3f3ba54698cc5f already in progress: creating tablet
I20250112 02:52:47.075748 3110 tablet_replica.cc:331] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:47.076373 3110 raft_consensus.cc:2238] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:47.077973 3110 raft_consensus.cc:2267] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:47.078909 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6151006e155440b7b02ab5e927805101 consensus state has no leader
I20250112 02:52:47.081087 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_a3d429eae14c4f6fbc5d9191a1a170f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.081544 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3d429eae14c4f6fbc5d9191a1a170f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.082854 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_a0ccfcee886d449d8bfd5bc1f9ca38df" } modify_external_catalogs: true
I20250112 02:52:47.083007 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_a0ccfcee886d449d8bfd5bc1f9ca38df" } modify_external_catalogs: true
I20250112 02:52:47.092479 450 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 79eb116c1eb44fb6b41129969f72f933
I20250112 02:52:47.104089 3109 tablet_replica.cc:331] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:47.104262 3138 consensus_queue.cc:1035] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.267s
I20250112 02:52:47.105470 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3771166b77e4f90aaa116ab1a9dc299" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:47.113294 452 master_service.cc:946] ReplaceTablet: received request to replace tablet f8c01b80fc8242d98fdfbf767212144d from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:47.114040 754 tablet_service.cc:1467] Processing CreateTablet for tablet 38e4a69f470a4d62b865e2dd594fab63 (DEFAULT_TABLE table=default.Table_c7dcf1defec042dabf093c4b9b0449a7 [id=77b9615977a74bceb6f208fb5a28742f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.114717 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d909275d67c4127b3144de53efc7d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.119410 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet f8c01b80fc8242d98fdfbf767212144d
I20250112 02:52:47.121923 3110 ts_tablet_manager.cc:1905] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:47.122355 3109 raft_consensus.cc:2238] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:47.122601 3109 pending_rounds.cc:62] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:47.122637 3109 pending_rounds.cc:69] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319883774734336 op_type: NO_OP noop_request { }
I20250112 02:52:47.122714 3109 raft_consensus.cc:2883] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:47.122773 3109 raft_consensus.cc:2267] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:47.123206 3109 ts_tablet_manager.cc:1905] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:47.080099 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 09900b42e8a443e680cc35c96a1dbbb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:47.081404 1181 raft_consensus.cc:357] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:47.123433 1181 raft_consensus.cc:383] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:47.123535 1181 raft_consensus.cc:738] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:47.124958 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_bb6854d0dab24b32a42819a3c18ac0d5" } modify_external_catalogs: true
I20250112 02:52:47.125157 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_bb6854d0dab24b32a42819a3c18ac0d5" } modify_external_catalogs: true
I20250112 02:52:47.125664 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_b910dc09eda7442ea8f6df224e60e7db" } modify_external_catalogs: true
I20250112 02:52:47.125751 744 tablet_service.cc:1467] Processing CreateTablet for tablet 1784b6e1337a440ebbae8eb885a51fca (DEFAULT_TABLE table=default.Table_c8d6f1dc529c44efa678ba41a8cde7e9 [id=376101d8628a4ffd9318fdc5f34b6c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.125813 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_b910dc09eda7442ea8f6df224e60e7db" } modify_external_catalogs: true
I20250112 02:52:47.126159 742 tablet_service.cc:1467] Processing CreateTablet for tablet 1bd1fb8d69bd411bb02fdafa6ba720f5 (DEFAULT_TABLE table=default.Table_6a2dccba50e94d42b01b4ad9e1ca5a35 [id=e8fe98863848407f97eb0c8c4d3c8235]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.126519 753 tablet_service.cc:1467] Processing CreateTablet for tablet b3392229868e4b08a16874f4a1b76de2 (DEFAULT_TABLE table=default.Table_7ca45e521bed4ddf8d26c9d32651eb34 [id=3f152d4fbcf3491e911f1a6716370f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.127976 751 tablet_service.cc:1467] Processing CreateTablet for tablet 51334a07328c4e7d92c843dd18907671 (DEFAULT_TABLE table=default.Table_f37babaaa3a241778a5a5542b9b832b0 [id=fcad044651084d92b3b57aa272cbedc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.129096 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_01fbb219edf24a889c196cc25eafc530" } modify_external_catalogs: true
I20250112 02:52:47.129271 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_01fbb219edf24a889c196cc25eafc530" } modify_external_catalogs: true
I20250112 02:52:47.138294 452 catalog_manager.cc:6596] ReplaceTablet: tablet f8c01b80fc8242d98fdfbf767212144d deleted and replaced by tablet 54b1182029af4678974df2a48c2a0dfd
I20250112 02:52:47.141216 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 5d0c3191973a477e949fbf9dfcca963f from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:47.143976 898 tablet_service.cc:1514] Processing DeleteTablet for tablet f8c01b80fc8242d98fdfbf767212144d with delete_type TABLET_DATA_DELETED (replaced by tablet 54b1182029af4678974df2a48c2a0dfd) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:47.147039 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 5d0c3191973a477e949fbf9dfcca963f
I20250112 02:52:47.147563 452 catalog_manager.cc:6596] ReplaceTablet: tablet 5d0c3191973a477e949fbf9dfcca963f deleted and replaced by tablet 4e6959ae2aca45fab60f936dab5c0f23
I20250112 02:52:47.148593 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d0c3191973a477e949fbf9dfcca963f with delete_type TABLET_DATA_DELETED (replaced by tablet 4e6959ae2aca45fab60f936dab5c0f23) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:47.148806 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 648960d4f08f4ce8a4f00aeeb34048f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:47.150068 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_330e355c75d94f3bbd8be0ca23f55b3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:47.150180 745 tablet_service.cc:1467] Processing CreateTablet for tablet 2473488c4890403291e38f287a976851 (DEFAULT_TABLE table=default.Table_7f1cd8fa7b844d0fba0cad8d1a6652e6 [id=cc8ce7ac6e4a416486ad1175ca1e1d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.150386 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 09900b42e8a443e680cc35c96a1dbbb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:47.150678 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_330e355c75d94f3bbd8be0ca23f55b3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.150710 750 tablet_service.cc:1467] Processing CreateTablet for tablet f01da4bb769843e5b81c0c6ba0ab9db2 (DEFAULT_TABLE table=default.Table_bfcd8a99635e43a29c4a91082cee18f4 [id=0332251275a74598ba6ee151af0481c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.151075 748 tablet_service.cc:1467] Processing CreateTablet for tablet a1ea1a1491a945049f9f4164eea7535c (DEFAULT_TABLE table=default.Table_83ec35a32e3e47fc9ccdde96b4f221df [id=ae920b41954f4968b81659c0b2a8e2cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.151448 759 tablet_service.cc:1467] Processing CreateTablet for tablet 1b34cef7c8a14dbda7c091dfed46515a (DEFAULT_TABLE table=default.Table_ea6f20bc46ec487cae9e3abf6419c80c [id=34602f2c95734d478e38e1847de46aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.151665 758 tablet_service.cc:1467] Processing CreateTablet for tablet ee0c474eedcd459697261747e802279e (DEFAULT_TABLE table=default.Table_65dfa36d9f664916bf107d10764f88cd [id=39a1a01693ae4665b769712b85ba5311]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.152184 743 tablet_service.cc:1467] Processing CreateTablet for tablet be44eccc90d24b5ea5f8fcfc8a74e039 (DEFAULT_TABLE table=default.Table_a12ca3f6249040c8bc457ee894d7a274 [id=3f439ae9d2ed4b009661a02117df21c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.152611 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_78a2b4e72b2f49aa95b57212aa2e418f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.153160 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78a2b4e72b2f49aa95b57212aa2e418f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.158178 757 tablet_service.cc:1467] Processing CreateTablet for tablet 26e2baf248444134a391242bad358ebe (DEFAULT_TABLE table=default.Table_468eef2886374c71b52269b9a47a4772 [id=0a2e5e706d9a4ca89f7620c3808dec66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.158334 741 tablet_service.cc:1467] Processing CreateTablet for tablet bbaa7dfc7da14cc2b4f3731d3da6107b (DEFAULT_TABLE table=default.Table_fb9557bec89748dabf949080f016deba [id=e0e08894993647c69bc6a75ec9f4bcbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.158452 752 tablet_service.cc:1514] Processing DeleteTablet for tablet a7db121c542f423bbffd2e36c459c3f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:47.158568 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 2796289c7e8f4b45b26508c50b57c640 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:47.158766 752 tablet_service.cc:1467] Processing CreateTablet for tablet 7d0d84297810420b8b7a4b24841eda55 (DEFAULT_TABLE table=default.Table_35eb917609ca4728bf4823fb912a9680 [id=80a612296e3742ee8906307445cc2ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.166972 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_c3cfd78c4fd14e4ca682ba5ae32499d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.167932 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3cfd78c4fd14e4ca682ba5ae32499d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.175065 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38e4a69f470a4d62b865e2dd594fab63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.176990 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bd1fb8d69bd411bb02fdafa6ba720f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.178283 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_945a8e7308c94078a007f67727d6a3bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.178807 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_945a8e7308c94078a007f67727d6a3bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.183737 755 tablet_service.cc:1514] Processing DeleteTablet for tablet b768b7534e034bff86045e8a474bf05d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:46 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:47.184043 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3392229868e4b08a16874f4a1b76de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.184366 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1784b6e1337a440ebbae8eb885a51fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.184572 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ea1a1491a945049f9f4164eea7535c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.184777 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be44eccc90d24b5ea5f8fcfc8a74e039. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.185333 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_35eb917609ca4728bf4823fb912a9680" } new_table_name: "default.Table_5ccb3f3bcf804cdb99da9d292491d7a2" modify_external_catalogs: true
I20250112 02:52:47.185464 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26e2baf248444134a391242bad358ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.185734 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbaa7dfc7da14cc2b4f3731d3da6107b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.186144 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee0c474eedcd459697261747e802279e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.186424 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2473488c4890403291e38f287a976851. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:47.188565 855 leader_election.cc:343] T d86417fc1c844cdf85323b94c6dce6d5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:47.188752 855 leader_election.cc:304] T d86417fc1c844cdf85323b94c6dce6d5 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:47.189950 3260 raft_consensus.cc:2743] T d86417fc1c844cdf85323b94c6dce6d5 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:47.192051 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d0d84297810420b8b7a4b24841eda55 consensus state has no leader
I20250112 02:52:47.201215 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_034a5a3b50f44a659402fc0321758528"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.201684 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_034a5a3b50f44a659402fc0321758528 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.203012 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b34cef7c8a14dbda7c091dfed46515a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.203220 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d0d84297810420b8b7a4b24841eda55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.203914 746 tablet_service.cc:1467] Processing CreateTablet for tablet 2329b3ddfae44b96a637ed00247f919d (DEFAULT_TABLE table=default.Table_84d9fd4fb68b4c31a7f438c12f67d042 [id=938d0e6585be408e8a842d9b9964c48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.204114 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_fe7d42a7f9a645048b94a8eb99cf431b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.204685 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe7d42a7f9a645048b94a8eb99cf431b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.206974 749 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea806f38475444c8f3d67e32bec9244 (DEFAULT_TABLE table=default.Table_43f7ba33caf249c4ba386a9199462957 [id=6a6c608131944f858e911d7f376f2813]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.208191 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_15bf41d2cd254fbc8ec6477f1b3deb55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.208752 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15bf41d2cd254fbc8ec6477f1b3deb55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.211275 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f01da4bb769843e5b81c0c6ba0ab9db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.211855 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51334a07328c4e7d92c843dd18907671. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.215154 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 058a9ca089cd4967887b0a1127c7d03f (table default.Table_a56064e2c6b449b7b1e32f2a69f2f7fe [id=448877a19b0d4ed2abbc3a796e40eaeb]) successfully deleted
I20250112 02:52:47.215415 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ea806f38475444c8f3d67e32bec9244. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:47.215759 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 76009bac22ce4136b148b493a004a0f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76009bac22ce4136b148b493a004a0f7 already in progress: creating tablet
W20250112 02:52:47.215996 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f3771166b77e4f90aaa116ab1a9dc299 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3771166b77e4f90aaa116ab1a9dc299 already in progress: creating tablet
W20250112 02:52:47.216176 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 940affdc2c484b7d9fd2de17f9d1cea7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 940affdc2c484b7d9fd2de17f9d1cea7 already in progress: creating tablet
W20250112 02:52:47.216363 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet bab4978edea0445fb0456e88e7fb4c44 because the tablet was not found. No further retry: Not found: Tablet not found: bab4978edea0445fb0456e88e7fb4c44
I20250112 02:52:47.216337 755 tablet_service.cc:1467] Processing CreateTablet for tablet b5485bdf54ab40b8b2b9b0bbac522468 (DEFAULT_TABLE table=default.Table_ef4fac0bfa544b4da7abc353a32d4e6b [id=56b2f9b180e744c4a079330bae08ac84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.220036 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2329b3ddfae44b96a637ed00247f919d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.220391 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5485bdf54ab40b8b2b9b0bbac522468. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.224875 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 0fea8f6f13d54945934ee3880e00c5e9
I20250112 02:52:47.069267 1173 tablet_bootstrap.cc:492] T 3499374810fb412e938325434c35618d P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:47.236706 1173 ts_tablet_manager.cc:1397] T 3499374810fb412e938325434c35618d P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.578s user 0.004s sys 0.025s
I20250112 02:52:47.072533 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca98a3f0fac4d2590c2c88aea34281e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.236860 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fae49da375f4c6a894c1237dd6621fb with delete_type TABLET_DATA_DELETED (replaced by tablet f203c86211f94999951740e1e6351b19) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:47.237913 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 4af0d07a878c42f5a15ff9b8a2295f30
I20250112 02:52:47.072746 3111 ts_tablet_manager.cc:1918] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:47.238233 3111 log.cc:1198] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/7b9c845571f44c8eb8753090d63df45f
I20250112 02:52:47.238585 3111 ts_tablet_manager.cc:1939] T 7b9c845571f44c8eb8753090d63df45f P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:47.239317 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 438) took 1512 ms. Trace:
I20250112 02:52:47.239372 3111 rpcz_store.cc:276] 0112 02:52:45.726752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.726801 (+ 49us) service_pool.cc:225] Handling call
0112 02:52:46.964964 (+1238163us) ts_tablet_manager.cc:1145] Deleting tablet 7b9c845571f44c8eb8753090d63df45f
0112 02:52:46.964972 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.019129 (+ 54157us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:47.072649 (+ 53520us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:47.239303 (+166654us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1237976}
I20250112 02:52:47.239622 3111 tablet_replica.cc:331] T ff1b3ee229c445b19405cf4a19cf8003 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:47.202528 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c54c5dc86446cf85974777c8145ef6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:47.239756 3111 raft_consensus.cc:2238] T ff1b3ee229c445b19405cf4a19cf8003 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:47.239861 3111 raft_consensus.cc:2267] T ff1b3ee229c445b19405cf4a19cf8003 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:47.240262 3111 ts_tablet_manager.cc:1905] T ff1b3ee229c445b19405cf4a19cf8003 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:47.123736 1181 consensus_queue.cc:260] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:47.241904 1181 ts_tablet_manager.cc:1428] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.203s user 0.002s sys 0.000s
I20250112 02:52:47.242182 1181 tablet_bootstrap.cc:492] T ca2165362ee541adac724b63f39b1489 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:47.075953 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fae49da375f4c6a894c1237dd6621fb with delete_type TABLET_DATA_DELETED (replaced by tablet f203c86211f94999951740e1e6351b19) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:47.244129 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_41b7056b6e9240928d0af52aec2e9338" } new_table_name: "default.Table_8ffa45da6c19440cbac5f82e79c6641b" modify_external_catalogs: true
I20250112 02:52:47.080605 2745 raft_consensus.cc:491] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:47.246006 2745 raft_consensus.cc:513] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:47.117627 991 leader_election.cc:343] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:47.216835 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c29a9ca43e034b43a00207e5847bbee3" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:47.246505 1055 raft_consensus.cc:3054] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:47.246526 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 128889us
I20250112 02:52:47.246584 3110 ts_tablet_manager.cc:1918] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:47.246690 3110 log.cc:1198] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/61eac6b572ff47c185768bee7490edf7
W20250112 02:52:47.246814 991 leader_election.cc:343] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:47.246902 991 leader_election.cc:304] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:47.246994 3110 ts_tablet_manager.cc:1939] T 61eac6b572ff47c185768bee7490edf7 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:47.247228 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 361) took 1873 ms. Trace:
I20250112 02:52:47.247293 3110 rpcz_store.cc:276] 0112 02:52:45.373583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.373654 (+ 71us) service_pool.cc:225] Handling call
0112 02:52:47.075708 (+1702054us) ts_tablet_manager.cc:1145] Deleting tablet 61eac6b572ff47c185768bee7490edf7
0112 02:52:47.075716 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.243610 (+167894us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:47.246520 (+ 2910us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:47.247216 (+ 696us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1701893}
I20250112 02:52:47.247565 3266 raft_consensus.cc:2743] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:47.247874 3110 tablet_replica.cc:331] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:47.248036 3110 raft_consensus.cc:2238] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:47.248286 3110 raft_consensus.cc:2267] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:47.248685 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet af689c1a3e2d41beb2199bf5d033aa29
I20250112 02:52:47.248764 3110 ts_tablet_manager.cc:1905] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:47.249414 3111 ts_tablet_manager.cc:1918] T ff1b3ee229c445b19405cf4a19cf8003 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:47.249505 3111 log.cc:1198] T ff1b3ee229c445b19405cf4a19cf8003 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/ff1b3ee229c445b19405cf4a19cf8003
I20250112 02:52:47.249652 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet bf2df39addb14d679b8cb7960a62d603
W20250112 02:52:47.252666 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca2165362ee541adac724b63f39b1489 consensus state has no leader
I20250112 02:52:47.254139 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_66ecd84e49d64f50b63b520f74b8e062" } new_table_name: "default.Table_9035b1f0824d486aa993f6bc318ed623" modify_external_catalogs: true
I20250112 02:52:47.261540 744 tablet_service.cc:1467] Processing CreateTablet for tablet efa6cc11872f4e67a9e7b73bd65651fa (DEFAULT_TABLE table=default.Table_b93a3e5a4b5c4739bc3153d38c77fabd [id=be97fe5bb0e7414f8efdffdf24465295]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.261853 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efa6cc11872f4e67a9e7b73bd65651fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.265942 3110 ts_tablet_manager.cc:1918] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:47.266098 3110 log.cc:1198] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/26577b614e32458e8bc5d6830c8c2e40
I20250112 02:52:47.266444 3110 ts_tablet_manager.cc:1939] T 26577b614e32458e8bc5d6830c8c2e40 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:47.266827 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 362) took 1866 ms. Trace:
I20250112 02:52:47.266945 3110 rpcz_store.cc:276] 0112 02:52:45.400629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.401107 (+ 478us) service_pool.cc:225] Handling call
0112 02:52:47.247836 (+1846729us) ts_tablet_manager.cc:1145] Deleting tablet 26577b614e32458e8bc5d6830c8c2e40
0112 02:52:47.247842 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.251279 (+ 3437us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:47.265869 (+ 14590us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:47.266813 (+ 944us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1846536}
I20250112 02:52:47.267441 3110 tablet_replica.cc:331] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:47.268107 3110 raft_consensus.cc:2238] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:47.282068 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_f2431b9445b84d008a9902efe7bbcd66" } new_table_name: "default.Table_d18ae031381c456ea6f576a3b343ffc4" modify_external_catalogs: true
I20250112 02:52:47.283725 741 tablet_service.cc:1467] Processing CreateTablet for tablet b1cb48c936844d049a1c832b47986d73 (DEFAULT_TABLE table=default.Table_5aeb2d7bb68c46a589a48526dcc40aaf [id=9d267774bf59428e854651956fbd79be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.283712 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_274f7d3306fd4ccd914f568e351f572a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:47.284132 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1cb48c936844d049a1c832b47986d73. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:47.284312 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_274f7d3306fd4ccd914f568e351f572a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.285864 755 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca98a3f0fac4d2590c2c88aea34281e (DEFAULT_TABLE table=default.Table_7077f9a718664d8b915ee4003502306a [id=6df53b1d0c7a4fa1a18b3202741630b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.286303 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca98a3f0fac4d2590c2c88aea34281e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.287153 3109 ts_tablet_manager.cc:1918] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:47.287282 3109 log.cc:1198] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/b1d9fae290ec4281839991f9d40b105c
I20250112 02:52:47.289173 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_5ad4c4df7d3944b898099c4681510214"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.289608 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ad4c4df7d3944b898099c4681510214 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.292938 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 61eac6b572ff47c185768bee7490edf7 (table default.Table_e8e62bd2fffe4f50849f722dbece2a2c [id=3603f999c77a48dca737bc8ff74eb9de]) successfully deleted
I20250112 02:52:47.293056 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_58df952efd74418bafb981fb49dfebee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:47.293285 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 26577b614e32458e8bc5d6830c8c2e40 (table default.Table_03e3448d809c4fc792d0bff94a713044 [id=5f3cc73b47414c438591107e35a12472]) successfully deleted
W20250112 02:52:47.293479 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58df952efd74418bafb981fb49dfebee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.294432 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_ce0ccdae99624243bdac1499da1790ee" } modify_external_catalogs: true
I20250112 02:52:47.294629 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_ce0ccdae99624243bdac1499da1790ee" } modify_external_catalogs: true
W20250112 02:52:47.298602 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f559dedfb7854cf890bef362c50b4213 consensus state has no leader
I20250112 02:52:47.299865 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 777272e7159b438ebff4db93698411d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 0925ba2197bb461190b832d062567040) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:47.300300 751 tablet_service.cc:1467] Processing CreateTablet for tablet a7db121c542f423bbffd2e36c459c3f0 (DEFAULT_TABLE table=default.Table_dc292752faf041ed8ca40958c86765b0 [id=407e1e8894524b7682f498b3904b06e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.300755 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7db121c542f423bbffd2e36c459c3f0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:47.303359 3175 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.022s sys 0.005s
W20250112 02:52:47.303440 3175 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s user 0.022s sys 0.005s
I20250112 02:52:47.305459 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 7b9c845571f44c8eb8753090d63df45f (table default.Table_7077f9a718664d8b915ee4003502306a [id=6df53b1d0c7a4fa1a18b3202741630b5]) successfully deleted
I20250112 02:52:47.305768 3159 raft_consensus.cc:491] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:47.305889 3159 raft_consensus.cc:513] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:47.306016 2745 leader_election.cc:290] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:47.306324 3111 ts_tablet_manager.cc:1939] T ff1b3ee229c445b19405cf4a19cf8003 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:47.310101 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet b8b64e4e64cc462891f1ee7a594d5026
I20250112 02:52:47.311295 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_9b65dd291c9f4fcdaa6d5e37f04d24ae" } new_table_name: "default.Table_b8a99487e7a2452aa61a90f4584e34b4" modify_external_catalogs: true
I20250112 02:52:47.315593 3159 leader_election.cc:290] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:47.316145 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 439) took 1400 ms. Trace:
I20250112 02:52:47.316243 3111 rpcz_store.cc:276] 0112 02:52:45.915689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.916302 (+ 613us) service_pool.cc:225] Handling call
0112 02:52:47.239591 (+1323289us) ts_tablet_manager.cc:1145] Deleting tablet ff1b3ee229c445b19405cf4a19cf8003
0112 02:52:47.239598 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.241932 (+ 2334us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:47.249342 (+ 7410us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:47.316125 (+ 66783us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1318269}
I20250112 02:52:47.316563 3111 tablet_replica.cc:331] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:47.317452 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fae49da375f4c6a894c1237dd6621fb with delete_type TABLET_DATA_DELETED (replaced by tablet f203c86211f94999951740e1e6351b19) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:47.317688 754 tablet_service.cc:1514] Processing DeleteTablet for tablet f8c01b80fc8242d98fdfbf767212144d with delete_type TABLET_DATA_DELETED (replaced by tablet 54b1182029af4678974df2a48c2a0dfd) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:47.317864 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d0c3191973a477e949fbf9dfcca963f with delete_type TABLET_DATA_DELETED (replaced by tablet 4e6959ae2aca45fab60f936dab5c0f23) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:47.318051 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fea8f6f13d54945934ee3880e00c5e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:47.318390 754 tablet_service.cc:1467] Processing CreateTablet for tablet a18c4709d4a845309968abafac18f58f (DEFAULT_TABLE table=default.Table_1ccecb4c9e1f481b9c467fea54894390 [id=d7bf75a59e0e4d4fa218633503f18668]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.318790 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a18c4709d4a845309968abafac18f58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.324285 3109 ts_tablet_manager.cc:1939] T b1d9fae290ec4281839991f9d40b105c P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
W20250112 02:52:47.292711 855 leader_election.cc:343] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:47.324808 855 leader_election.cc:304] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:47.324856 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet f8c01b80fc8242d98fdfbf767212144d with delete_type TABLET_DATA_DELETED (replaced by tablet 54b1182029af4678974df2a48c2a0dfd) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:47.325619 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d0c3191973a477e949fbf9dfcca963f with delete_type TABLET_DATA_DELETED (replaced by tablet 4e6959ae2aca45fab60f936dab5c0f23) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:47.325719 892 tablet_service.cc:1467] Processing CreateTablet for tablet 05bc6712c15a45d893f2676bad6eba71 (DEFAULT_TABLE table=default.Table_e968295bfd5546dfb74184d506b542e8 [id=d7c723b1352c4b4d83e1a3a5c2383238]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.326115 895 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3de545ebcb45f6bb72b97126893b44 (DEFAULT_TABLE table=default.Table_8c1228cf5d9a4355811e6430527d39b2 [id=9df3ad6a24044a0eb5eab20369cd6631]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.326378 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c3de545ebcb45f6bb72b97126893b44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.329528 897 tablet_service.cc:1467] Processing CreateTablet for tablet a18c4709d4a845309968abafac18f58f (DEFAULT_TABLE table=default.Table_1ccecb4c9e1f481b9c467fea54894390 [id=d7bf75a59e0e4d4fa218633503f18668]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.329927 3000 raft_consensus.cc:2743] T 17c54c5dc86446cf85974777c8145ef6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:47.308545 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fea8f6f13d54945934ee3880e00c5e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:47.330188 1173 raft_consensus.cc:357] T 3499374810fb412e938325434c35618d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:47.330418 1173 raft_consensus.cc:383] T 3499374810fb412e938325434c35618d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:47.330466 1173 raft_consensus.cc:738] T 3499374810fb412e938325434c35618d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:47.330653 3110 raft_consensus.cc:2267] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:47.330682 1173 consensus_queue.cc:260] T 3499374810fb412e938325434c35618d P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:47.331219 1173 ts_tablet_manager.cc:1428] T 3499374810fb412e938325434c35618d P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.094s user 0.000s sys 0.000s
I20250112 02:52:47.331418 3110 ts_tablet_manager.cc:1905] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:47.331532 1173 tablet_bootstrap.cc:492] T ea33053c62dc4707b759d5c1756ab4bd P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:47.338266 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fea8f6f13d54945934ee3880e00c5e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:47.338603 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 05bc6712c15a45d893f2676bad6eba71 (DEFAULT_TABLE table=default.Table_e968295bfd5546dfb74184d506b542e8 [id=d7c723b1352c4b4d83e1a3a5c2383238]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.338928 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05bc6712c15a45d893f2676bad6eba71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.339190 1025 tablet_service.cc:1467] Processing CreateTablet for tablet af689c1a3e2d41beb2199bf5d033aa29 (DEFAULT_TABLE table=default.Table_bb6854d0dab24b32a42819a3c18ac0d5 [id=c88e676d22964c3e900d903c4237a7e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.339366 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 56a5e2eadfa24c8da83706f05b385ae8 (DEFAULT_TABLE table=default.Table_5d4ae07cad674fa4a7d2253d5eccaa5b [id=61120a8fc1144f29a0eaa037a2d47554]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.339519 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af689c1a3e2d41beb2199bf5d033aa29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.339635 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56a5e2eadfa24c8da83706f05b385ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.340873 1031 tablet_service.cc:1467] Processing CreateTablet for tablet bf2df39addb14d679b8cb7960a62d603 (DEFAULT_TABLE table=default.Table_01fbb219edf24a889c196cc25eafc530 [id=ff3a549238d741b0ac6fcbb78199c589]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.341127 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf2df39addb14d679b8cb7960a62d603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.342702 1033 tablet_service.cc:1467] Processing CreateTablet for tablet b664fabebd844f2ab744755ceb150511 (DEFAULT_TABLE table=default.Table_a5b8c2cd15dd47818378fc3a3065b890 [id=3501de8bb4f04582af6b7029ab52c9b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.342957 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b664fabebd844f2ab744755ceb150511. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.343716 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef34ffff8f24f819d7f1cfafdc639de (DEFAULT_TABLE table=default.Table_1950cb8738a5407a84000b99c2cf5e2e [id=16d678639878422ba4df5abbe6dc0a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.343912 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 9af4a0d53030406a968b81ca5989303f (DEFAULT_TABLE table=default.Table_872ea90a0f274baea3d3d36c306f2e35 [id=cb61e5f229d14548a1fe778b731a4c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.344106 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3de545ebcb45f6bb72b97126893b44 (DEFAULT_TABLE table=default.Table_8c1228cf5d9a4355811e6430527d39b2 [id=9df3ad6a24044a0eb5eab20369cd6631]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.344300 1016 tablet_service.cc:1467] Processing CreateTablet for tablet a2e2a64032f84fadb05f4455c2c51529 (DEFAULT_TABLE table=default.Table_866e2b7327c64126af36d106ef79ce17 [id=d020634d12fe45cfac5dbada38a37a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.344491 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 65e6d77ae6ab422a9e1c9a8e1e13183d (DEFAULT_TABLE table=default.Table_9c23aecb8f9240049865c0e9db0f25b7 [id=7c7bc81bff854be4af7199b0d4c39d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.344820 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 475b8ea7d68b44fba7431a2812c13871 (DEFAULT_TABLE table=default.Table_b4223b0bf2f348158ec6104d6da3ba42 [id=7f55f14b50cb4e1ea2575cd054539670]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.345403 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet bf2df39addb14d679b8cb7960a62d603 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:47.345618 1021 tablet_service.cc:1467] Processing CreateTablet for tablet a8459f094b6e40ea9b5f688f937a9d7d (DEFAULT_TABLE table=default.Table_e44403a267f048b6aefc3ae0848fbab8 [id=6c2b160570ac48e9af2014c2de88e92c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.345837 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b64e4e64cc462891f1ee7a594d5026 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:47.343732 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 0925ba2197bb461190b832d062567040 (DEFAULT_TABLE table=default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.345964 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 0f58a14e35eb41e591afce5a1159c985 (DEFAULT_TABLE table=default.Table_75a41a4f8be84a56b1103d08e0510d4a [id=b36674fcee61486cad9eb7c5c1338c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.346182 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 4af0d07a878c42f5a15ff9b8a2295f30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:47.346426 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 436eecfe1f34455ead6bad2b884a7cee (DEFAULT_TABLE table=default.Table_ee86b03e71a1459f8fbf833684f9b321 [id=25e4d7a8a23e4603af050b5b1a532375]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.346696 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet af689c1a3e2d41beb2199bf5d033aa29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:47.347118 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet ff1b3ee229c445b19405cf4a19cf8003 (table default.Table_6b8cb70a5e5b4519af5d247f6b5bb8b0 [id=a28f5cfbcfe64768b40477ffc2053e27]) successfully deleted
W20250112 02:52:47.353302 3144 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.004s sys 0.027s
W20250112 02:52:47.353449 3144 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s user 0.004s sys 0.027s
I20250112 02:52:47.359752 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_f2259d9cdec345a1923599bd9b188e92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.360263 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2259d9cdec345a1923599bd9b188e92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.363246 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_ee4d7e840acc457c88cd3b4de3824bbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.363703 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee4d7e840acc457c88cd3b4de3824bbe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.370980 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05bc6712c15a45d893f2676bad6eba71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.371202 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a18c4709d4a845309968abafac18f58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.375478 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_1b6e075c7c0a4937ac65ce1bebdc1af6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.375981 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b6e075c7c0a4937ac65ce1bebdc1af6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.382453 893 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef34ffff8f24f819d7f1cfafdc639de (DEFAULT_TABLE table=default.Table_1950cb8738a5407a84000b99c2cf5e2e [id=16d678639878422ba4df5abbe6dc0a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.382786 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef34ffff8f24f819d7f1cfafdc639de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.382753 1022 tablet_service.cc:1467] Processing CreateTablet for tablet a18c4709d4a845309968abafac18f58f (DEFAULT_TABLE table=default.Table_1ccecb4c9e1f481b9c467fea54894390 [id=d7bf75a59e0e4d4fa218633503f18668]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.386847 897 tablet_service.cc:1467] Processing CreateTablet for tablet 65e6d77ae6ab422a9e1c9a8e1e13183d (DEFAULT_TABLE table=default.Table_9c23aecb8f9240049865c0e9db0f25b7 [id=7c7bc81bff854be4af7199b0d4c39d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.387159 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e6d77ae6ab422a9e1c9a8e1e13183d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.387391 1055 raft_consensus.cc:2463] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:47.387661 3111 raft_consensus.cc:2238] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:47.389891 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 38455a4120f14388b55d94e7fc525c47 (DEFAULT_TABLE table=default.Table_542b9b0705ee44849946cbbb857a5f9f [id=c05ca36543d444b7a99f91308aeea3ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.390646 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_ac3a92da0f3e4df19577758643bfb95d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.391156 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac3a92da0f3e4df19577758643bfb95d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.394173 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_525750dad2b7461586d070dbaa74d0b1" } new_table_name: "default.Table_8fd4122d870a475baea2cc3cb97bd01a" modify_external_catalogs: true
I20250112 02:52:47.394416 1181 tablet_bootstrap.cc:654] T ca2165362ee541adac724b63f39b1489 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:47.391296 3111 raft_consensus.cc:2267] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:47.402830 3111 ts_tablet_manager.cc:1905] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:47.396662 1173 tablet_bootstrap.cc:654] T ea33053c62dc4707b759d5c1756ab4bd P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:47.400434 2745 consensus_queue.cc:1035] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:47.418212 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 445) took 2508 ms. Trace:
I20250112 02:52:47.418545 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_8a8adb34395948c991016903d78996cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:47.419225 757 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef34ffff8f24f819d7f1cfafdc639de (DEFAULT_TABLE table=default.Table_1950cb8738a5407a84000b99c2cf5e2e [id=16d678639878422ba4df5abbe6dc0a80]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:47.419682 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a8adb34395948c991016903d78996cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.419735 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef34ffff8f24f819d7f1cfafdc639de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.420322 753 tablet_service.cc:1467] Processing CreateTablet for tablet 65e6d77ae6ab422a9e1c9a8e1e13183d (DEFAULT_TABLE table=default.Table_9c23aecb8f9240049865c0e9db0f25b7 [id=7c7bc81bff854be4af7199b0d4c39d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.421116 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_d00b7d11e6444f6192bb5a9687f1c589"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.421607 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d00b7d11e6444f6192bb5a9687f1c589 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.422070 748 tablet_service.cc:1467] Processing CreateTablet for tablet af689c1a3e2d41beb2199bf5d033aa29 (DEFAULT_TABLE table=default.Table_bb6854d0dab24b32a42819a3c18ac0d5 [id=c88e676d22964c3e900d903c4237a7e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.422489 743 tablet_service.cc:1467] Processing CreateTablet for tablet 05bc6712c15a45d893f2676bad6eba71 (DEFAULT_TABLE table=default.Table_e968295bfd5546dfb74184d506b542e8 [id=d7c723b1352c4b4d83e1a3a5c2383238]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.422991 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e6d77ae6ab422a9e1c9a8e1e13183d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:47.423027 1177 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.684s user 0.012s sys 0.020s
I20250112 02:52:47.423491 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05bc6712c15a45d893f2676bad6eba71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.424995 759 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3de545ebcb45f6bb72b97126893b44 (DEFAULT_TABLE table=default.Table_8c1228cf5d9a4355811e6430527d39b2 [id=9df3ad6a24044a0eb5eab20369cd6631]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.418371 3109 rpcz_store.cc:276] 0112 02:52:44.909265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.909412 (+ 147us) service_pool.cc:225] Handling call
0112 02:52:46.837570 (+1928158us) ts_tablet_manager.cc:1145] Deleting tablet b1d9fae290ec4281839991f9d40b105c
0112 02:52:46.837582 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.277194 (+439612us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:47.287091 (+ 9897us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:47.418189 (+131098us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18031,"tablet-delete.queue_time_us":1917041}
I20250112 02:52:47.427088 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af689c1a3e2d41beb2199bf5d033aa29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.428287 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_872ea90a0f274baea3d3d36c306f2e35" } modify_external_catalogs: true
I20250112 02:52:47.428500 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_872ea90a0f274baea3d3d36c306f2e35" } modify_external_catalogs: true
I20250112 02:52:47.428714 742 tablet_service.cc:1467] Processing CreateTablet for tablet 38455a4120f14388b55d94e7fc525c47 (DEFAULT_TABLE table=default.Table_542b9b0705ee44849946cbbb857a5f9f [id=c05ca36543d444b7a99f91308aeea3ae]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:47.423274 1177 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.685s user 0.012s sys 0.020s
I20250112 02:52:47.431905 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_fbceddd1104f4f24a7e21f0351add342"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.432462 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fbceddd1104f4f24a7e21f0351add342 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.432794 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 446) took 2508 ms. Trace:
I20250112 02:52:47.423293 745 tablet_service.cc:1467] Processing CreateTablet for tablet 475b8ea7d68b44fba7431a2812c13871 (DEFAULT_TABLE table=default.Table_b4223b0bf2f348158ec6104d6da3ba42 [id=7f55f14b50cb4e1ea2575cd054539670]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.437906 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38455a4120f14388b55d94e7fc525c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.438797 3000 raft_consensus.cc:491] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:47.438930 3000 raft_consensus.cc:513] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:47.439375 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_67e40ef8ee824430b5d090ddc9cc03c8" } modify_external_catalogs: true
I20250112 02:52:47.439460 3000 leader_election.cc:290] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:47.439545 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_67e40ef8ee824430b5d090ddc9cc03c8" } modify_external_catalogs: true
I20250112 02:52:47.432914 3109 rpcz_store.cc:276] 0112 02:52:44.924773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:44.924858 (+ 85us) service_pool.cc:225] Handling call
0112 02:52:47.425900 (+2501042us) ts_tablet_manager.cc:1145] Deleting tablet 93aba9b4d1dd485cb62c4827c916a1d5
0112 02:52:47.425908 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.432774 (+ 6866us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2500851}
I20250112 02:52:47.440622 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 9af4a0d53030406a968b81ca5989303f
I20250112 02:52:47.444171 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 456) took 2269 ms. Trace:
I20250112 02:52:47.298892 746 tablet_service.cc:1467] Processing CreateTablet for tablet 5b78e2f2303348dea1ca11c1cfd24e3f (DEFAULT_TABLE table=default.Table_8bb2bb131d6b4262b964926902af01d3 [id=8128f3bba480478993bc9b24e1d16c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.446230 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b78e2f2303348dea1ca11c1cfd24e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.446813 750 tablet_service.cc:1467] Processing CreateTablet for tablet bf2df39addb14d679b8cb7960a62d603 (DEFAULT_TABLE table=default.Table_01fbb219edf24a889c196cc25eafc530 [id=ff3a549238d741b0ac6fcbb78199c589]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.450388 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 500f0c5c76e54cd8808e6c8611e99d83
I20250112 02:52:47.451694 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475b8ea7d68b44fba7431a2812c13871. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.452339 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_76e4d0c54155412296bc900e03cd40bd" } modify_external_catalogs: true
I20250112 02:52:47.452450 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_8dde1cf3a26244709cf38e8563b97a2a" } modify_external_catalogs: true
I20250112 02:52:47.452523 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_76e4d0c54155412296bc900e03cd40bd" } modify_external_catalogs: true
I20250112 02:52:47.452591 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_8dde1cf3a26244709cf38e8563b97a2a" } modify_external_catalogs: true
I20250112 02:52:47.444304 3109 rpcz_store.cc:276] 0112 02:52:45.174282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.174503 (+ 221us) service_pool.cc:225] Handling call
0112 02:52:47.440110 (+2265607us) ts_tablet_manager.cc:1145] Deleting tablet 2361bc6bd8164c2daa37be6b0845f4bd
0112 02:52:47.440140 (+ 30us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.444156 (+ 4016us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2265310}
I20250112 02:52:47.454772 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_330e355c75d94f3bbd8be0ca23f55b3e" } new_table_name: "default.Table_7048a3b184fc4f85be97bc0227107539" modify_external_catalogs: true
I20250112 02:52:47.456466 756 tablet_service.cc:1467] Processing CreateTablet for tablet 56a5e2eadfa24c8da83706f05b385ae8 (DEFAULT_TABLE table=default.Table_5d4ae07cad674fa4a7d2253d5eccaa5b [id=61120a8fc1144f29a0eaa037a2d47554]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.462067 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_074b886d23664673a2a482c11ba59f15" } new_table_name: "default.Table_1c6251f36531494a9acda9cbe5b972d1" modify_external_catalogs: true
I20250112 02:52:47.462440 895 tablet_service.cc:1467] Processing CreateTablet for tablet 475b8ea7d68b44fba7431a2812c13871 (DEFAULT_TABLE table=default.Table_b4223b0bf2f348158ec6104d6da3ba42 [id=7f55f14b50cb4e1ea2575cd054539670]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.462765 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475b8ea7d68b44fba7431a2812c13871. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.462440 894 tablet_service.cc:1467] Processing CreateTablet for tablet 38455a4120f14388b55d94e7fc525c47 (DEFAULT_TABLE table=default.Table_542b9b0705ee44849946cbbb857a5f9f [id=c05ca36543d444b7a99f91308aeea3ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.462579 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 781fe3514f4c4f80b1ee49278502ccc1
I20250112 02:52:47.463112 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38455a4120f14388b55d94e7fc525c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.464475 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet f950c604609042da96e17398f42c1929
I20250112 02:52:47.465869 749 tablet_service.cc:1467] Processing CreateTablet for tablet a8459f094b6e40ea9b5f688f937a9d7d (DEFAULT_TABLE table=default.Table_e44403a267f048b6aefc3ae0848fbab8 [id=6c2b160570ac48e9af2014c2de88e92c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.467700 889 tablet_service.cc:1467] Processing CreateTablet for tablet af689c1a3e2d41beb2199bf5d033aa29 (DEFAULT_TABLE table=default.Table_bb6854d0dab24b32a42819a3c18ac0d5 [id=c88e676d22964c3e900d903c4237a7e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.468035 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af689c1a3e2d41beb2199bf5d033aa29. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:47.469550 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ff7a7cc250c446db0985ffc1bcc2f91 has no consensus state
I20250112 02:52:47.469530 758 tablet_service.cc:1467] Processing CreateTablet for tablet 0925ba2197bb461190b832d062567040 (DEFAULT_TABLE table=default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.470181 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0925ba2197bb461190b832d062567040. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.472589 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c3de545ebcb45f6bb72b97126893b44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.473201 880 tablet_service.cc:1467] Processing CreateTablet for tablet 0925ba2197bb461190b832d062567040 (DEFAULT_TABLE table=default.Table_f180982a92d4460d94f3f67eccd27009 [id=a842736799014e0886c155171bb11a13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.473520 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0925ba2197bb461190b832d062567040. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.473989 884 tablet_service.cc:1514] Processing DeleteTablet for tablet bf2df39addb14d679b8cb7960a62d603 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:38786
W20250112 02:52:47.474061 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 03276edd7bbe42b5a8ca16c5ccf1b337 consensus state has no leader
I20250112 02:52:47.474555 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af4a0d53030406a968b81ca5989303f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:47.474824 884 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4a41baf3ad44ffb9be6cd8c526bbcd (DEFAULT_TABLE table=default.Table_2fb335949dc24790a343c60f455a560a [id=c2b1663a1d674d8f8e5d37eeb58fb86c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.475133 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4a41baf3ad44ffb9be6cd8c526bbcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.475711 899 tablet_service.cc:1467] Processing CreateTablet for tablet 56a5e2eadfa24c8da83706f05b385ae8 (DEFAULT_TABLE table=default.Table_5d4ae07cad674fa4a7d2253d5eccaa5b [id=61120a8fc1144f29a0eaa037a2d47554]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.476007 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56a5e2eadfa24c8da83706f05b385ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.476822 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf2df39addb14d679b8cb7960a62d603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.476789 898 tablet_service.cc:1467] Processing CreateTablet for tablet a8459f094b6e40ea9b5f688f937a9d7d (DEFAULT_TABLE table=default.Table_e44403a267f048b6aefc3ae0848fbab8 [id=6c2b160570ac48e9af2014c2de88e92c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.477082 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8459f094b6e40ea9b5f688f937a9d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.477272 1177 tablet_bootstrap.cc:492] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:47.477383 1177 ts_tablet_manager.cc:1397] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.958s user 0.014s sys 0.020s
I20250112 02:52:47.477903 890 tablet_service.cc:1467] Processing CreateTablet for tablet b664fabebd844f2ab744755ceb150511 (DEFAULT_TABLE table=default.Table_a5b8c2cd15dd47818378fc3a3065b890 [id=3501de8bb4f04582af6b7029ab52c9b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.478163 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b664fabebd844f2ab744755ceb150511. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.479034 887 tablet_service.cc:1467] Processing CreateTablet for tablet 9af4a0d53030406a968b81ca5989303f (DEFAULT_TABLE table=default.Table_872ea90a0f274baea3d3d36c306f2e35 [id=cb61e5f229d14548a1fe778b731a4c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.479285 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af4a0d53030406a968b81ca5989303f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.479511 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 454) took 2305 ms. Trace:
I20250112 02:52:47.479583 3109 rpcz_store.cc:276] 0112 02:52:45.174111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.174483 (+ 372us) service_pool.cc:225] Handling call
0112 02:52:47.453149 (+2278666us) ts_tablet_manager.cc:1145] Deleting tablet 1a7717cdcb8a4f47b11ba7af54b57e3c
0112 02:52:47.453159 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.479494 (+ 26335us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2277710}
I20250112 02:52:47.480125 891 tablet_service.cc:1467] Processing CreateTablet for tablet 0f58a14e35eb41e591afce5a1159c985 (DEFAULT_TABLE table=default.Table_75a41a4f8be84a56b1103d08e0510d4a [id=b36674fcee61486cad9eb7c5c1338c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.480371 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f58a14e35eb41e591afce5a1159c985. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.480996 881 tablet_service.cc:1467] Processing CreateTablet for tablet a2e2a64032f84fadb05f4455c2c51529 (DEFAULT_TABLE table=default.Table_866e2b7327c64126af36d106ef79ce17 [id=d020634d12fe45cfac5dbada38a37a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.481210 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 4af0d07a878c42f5a15ff9b8a2295f30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:47.481464 882 tablet_service.cc:1467] Processing CreateTablet for tablet fc2b984178a446d193a9dec4dafc64e8 (DEFAULT_TABLE table=default.Table_b7da5ba607bd47e89d807929a64e2ea7 [id=8a0707ea3b06473d95af04bcd7384ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.481819 754 tablet_service.cc:1467] Processing CreateTablet for tablet b664fabebd844f2ab744755ceb150511 (DEFAULT_TABLE table=default.Table_a5b8c2cd15dd47818378fc3a3065b890 [id=3501de8bb4f04582af6b7029ab52c9b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.481819 883 tablet_service.cc:1467] Processing CreateTablet for tablet 436eecfe1f34455ead6bad2b884a7cee (DEFAULT_TABLE table=default.Table_ee86b03e71a1459f8fbf833684f9b321 [id=25e4d7a8a23e4603af050b5b1a532375]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.482100 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b664fabebd844f2ab744755ceb150511. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.482086 896 tablet_service.cc:1514] Processing DeleteTablet for tablet af689c1a3e2d41beb2199bf5d033aa29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:47.482371 896 tablet_service.cc:1467] Processing CreateTablet for tablet 08c1a4ccdb664c05a9999a9843ded765 (DEFAULT_TABLE table=default.Table_36b06d2edb8e4d23a3cb7021a8267db4 [id=3506f201d2674e38a353c87e1b2f26a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.482672 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08c1a4ccdb664c05a9999a9843ded765. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.483292 888 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b64e4e64cc462891f1ee7a594d5026 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:47.483506 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 500f0c5c76e54cd8808e6c8611e99d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:47.483840 888 tablet_service.cc:1467] Processing CreateTablet for tablet 1c48318e40954706b2c14a4573b5f5b8 (DEFAULT_TABLE table=default.Table_a1e87ce18198488dac5e9cbbb10fd42e [id=1e6e1e6b5db548c49c3872677fcae8be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.484150 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c48318e40954706b2c14a4573b5f5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.484844 885 tablet_service.cc:1467] Processing CreateTablet for tablet a94cec0f0d4d4f06aa5667135efbf5a6 (DEFAULT_TABLE table=default.Table_d3eb6d686d9540ffbdc854d3932aa893 [id=d1fa33e075d74f1193fe708f5034fbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.485095 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a94cec0f0d4d4f06aa5667135efbf5a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.485867 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56a5e2eadfa24c8da83706f05b385ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.494077 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 461) took 2317 ms. Trace:
I20250112 02:52:47.494233 3109 rpcz_store.cc:276] 0112 02:52:45.176390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.176435 (+ 45us) service_pool.cc:225] Handling call
0112 02:52:47.479784 (+2303349us) ts_tablet_manager.cc:1145] Deleting tablet a4328137407843e1879493ef0c56ffc8
0112 02:52:47.479793 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.494056 (+ 14263us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2303213}
I20250112 02:52:47.497145 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8459f094b6e40ea9b5f688f937a9d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.499249 886 tablet_service.cc:1467] Processing CreateTablet for tablet bf2df39addb14d679b8cb7960a62d603 (DEFAULT_TABLE table=default.Table_01fbb219edf24a889c196cc25eafc530 [id=ff3a549238d741b0ac6fcbb78199c589]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.499562 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf2df39addb14d679b8cb7960a62d603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.500123 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_f69a79e767cb4d6da20e31ddb7418a63" } new_table_name: "default.Table_d9117b8cf65240dab1d9c1f61fe05192" modify_external_catalogs: true
I20250112 02:52:47.504473 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_83f317260ae141bdba6ce160f19270fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.505035 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83f317260ae141bdba6ce160f19270fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.506512 741 tablet_service.cc:1467] Processing CreateTablet for tablet 9af4a0d53030406a968b81ca5989303f (DEFAULT_TABLE table=default.Table_872ea90a0f274baea3d3d36c306f2e35 [id=cb61e5f229d14548a1fe778b731a4c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.506877 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af4a0d53030406a968b81ca5989303f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.507623 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_36ea26af7c544341a2862484b5a9a881"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.508097 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36ea26af7c544341a2862484b5a9a881 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.514760 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_6e552be8f8b7468f84f7675b3342125e" } modify_external_catalogs: true
I20250112 02:52:47.514930 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_6e552be8f8b7468f84f7675b3342125e" } modify_external_catalogs: true
I20250112 02:52:47.515331 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d418cda9ed5449fa45690130cd6d5bc" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:47.515542 909 raft_consensus.cc:2463] T 3d418cda9ed5449fa45690130cd6d5bc P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:47.515758 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_26e7eaf3c8dd4451b39f683c627fbd21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.516207 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26e7eaf3c8dd4451b39f683c627fbd21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.516515 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 460) took 2340 ms. Trace:
I20250112 02:52:47.516592 3109 rpcz_store.cc:276] 0112 02:52:45.176303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.176589 (+ 286us) service_pool.cc:225] Handling call
0112 02:52:47.494566 (+2317977us) ts_tablet_manager.cc:1145] Deleting tablet ea33053c62dc4707b759d5c1756ab4bd
0112 02:52:47.494573 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.516495 (+ 21922us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2317871}
I20250112 02:52:47.516919 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 458) took 2342 ms. Trace:
I20250112 02:52:47.528947 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_fe00ca2755c7455f8b6c4629368d8a26" } new_table_name: "default.Table_637a2c39cc7c474598f1acda5fd1b3bb" modify_external_catalogs: true
I20250112 02:52:47.533131 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 2bf8a907f0af4e24819d1f00e40885ee
W20250112 02:52:47.524147 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 276833us
W20250112 02:52:47.543840 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ff848f9d33e74648a00255c92f599be5 consensus state has no leader
I20250112 02:52:47.546411 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fea8f6f13d54945934ee3880e00c5e9" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:47.516992 3109 rpcz_store.cc:276] 0112 02:52:45.174438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.174771 (+ 333us) service_pool.cc:225] Handling call
0112 02:52:47.516821 (+2342050us) ts_tablet_manager.cc:1145] Deleting tablet 4cfe1a1595014c27bbd350cec169d71a
0112 02:52:47.516830 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.516905 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2339994}
I20250112 02:52:47.550925 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 462) took 2370 ms. Trace:
I20250112 02:52:47.551084 3109 rpcz_store.cc:276] 0112 02:52:45.180208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.180255 (+ 47us) service_pool.cc:225] Handling call
0112 02:52:47.550812 (+2370557us) ts_tablet_manager.cc:1145] Deleting tablet f52b838214c54ac69ad30dac6e56669a
0112 02:52:47.550823 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.550908 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2370390}
I20250112 02:52:47.551458 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 463) took 2371 ms. Trace:
I20250112 02:52:47.553695 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 781fe3514f4c4f80b1ee49278502ccc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:47.561530 1177 raft_consensus.cc:357] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:47.563176 743 tablet_service.cc:1467] Processing CreateTablet for tablet 0f58a14e35eb41e591afce5a1159c985 (DEFAULT_TABLE table=default.Table_75a41a4f8be84a56b1103d08e0510d4a [id=b36674fcee61486cad9eb7c5c1338c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.563657 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f58a14e35eb41e591afce5a1159c985. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.551609 3109 rpcz_store.cc:276] 0112 02:52:45.180302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.180353 (+ 51us) service_pool.cc:225] Handling call
0112 02:52:47.551364 (+2371011us) ts_tablet_manager.cc:1145] Deleting tablet d95f98ae7bdf42518bbff55e257fea88
0112 02:52:47.551373 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.551446 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2370882}
I20250112 02:52:47.564340 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 464) took 2366 ms. Trace:
I20250112 02:52:47.564776 3109 rpcz_store.cc:276] 0112 02:52:45.197666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.197717 (+ 51us) service_pool.cc:225] Handling call
0112 02:52:47.564247 (+2366530us) ts_tablet_manager.cc:1145] Deleting tablet 64b0d58af4cf4adcbeb3ef5070f2982d
0112 02:52:47.564255 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.564326 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2366335}
I20250112 02:52:47.565176 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 465) took 2336 ms. Trace:
I20250112 02:52:47.565292 3109 rpcz_store.cc:276] 0112 02:52:45.228525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.231507 (+ 2982us) service_pool.cc:225] Handling call
0112 02:52:47.565087 (+2333580us) ts_tablet_manager.cc:1145] Deleting tablet be856e4b913e40d58a3f3ba54698cc5f
0112 02:52:47.565095 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.565163 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2333394}
I20250112 02:52:47.565610 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 466) took 2317 ms. Trace:
I20250112 02:52:47.571195 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_6b747ebd44244031a16f8b805caa9209"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.571734 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b747ebd44244031a16f8b805caa9209 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:47.571748 911 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.009s sys 0.006s
W20250112 02:52:47.571828 911 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.506s user 0.009s sys 0.006s
W20250112 02:52:47.572435 3188 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.702s user 0.009s sys 0.022s
W20250112 02:52:47.572516 3188 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s user 0.009s sys 0.022s
I20250112 02:52:47.573035 938 tablet_copy_source_session.cc:215] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 02:52:47.574079 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fea8f6f13d54945934ee3880e00c5e9" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:47.575521 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_ef6824526bd54a068a32b419534d46e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.575999 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef6824526bd54a068a32b419534d46e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.579006 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af4a0d53030406a968b81ca5989303f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:47.580354 854 leader_election.cc:343] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:47.580648 3273 consensus_queue.cc:1035] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:52:47.582525 918 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.731s user 0.011s sys 0.013s
W20250112 02:52:47.582602 918 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.732s user 0.011s sys 0.013s
I20250112 02:52:47.585520 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_f5445f2c41c740a5bf3ceb49b8491359"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.586232 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5445f2c41c740a5bf3ceb49b8491359 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:47.588546 2977 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.656s user 0.011s sys 0.005s
W20250112 02:52:47.588636 2977 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s user 0.011s sys 0.005s
I20250112 02:52:47.590574 3310 ts_tablet_manager.cc:927] T 5804fc4130f84c69832c1022b7bd7fa5 P f1e775296eb746db8b318f17d26820e6: Initiating tablet copy from peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
W20250112 02:52:47.590552 855 leader_election.cc:343] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:47.590652 855 leader_election.cc:304] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:47.591004 3260 raft_consensus.cc:2743] T 0fea8f6f13d54945934ee3880e00c5e9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:47.590803 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:47.591425 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_d9117b8cf65240dab1d9c1f61fe05192 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 2 got 0
I20250112 02:52:47.591681 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:47.591883 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ca6f080e5fd84084b601b646d4e6c9e2 (table default.Table_bb0a4f973f234c80954ac449b0c6e88a [id=f95ac050a4b840a19a8175caade87d48]). Expected version 3 got 0
I20250112 02:52:47.591861 892 tablet_service.cc:1514] Processing DeleteTablet for tablet f950c604609042da96e17398f42c1929 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:47.592301 446 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet b455a37448a044679bd2c0646de3d804 (table default.Table_9ecfcdf0c66d4e3184fae185f8fe6c9e [id=55eb92db6e924911b44fb44221d453e3]). Expected version 1 got 0
I20250112 02:52:47.595858 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bf8a907f0af4e24819d1f00e40885ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:47.599350 3307 ts_tablet_manager.cc:927] T b7340a7a916c4b2fab32e0ba2d0ce322 P f1e775296eb746db8b318f17d26820e6: Initiating tablet copy from peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:47.604735 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2e2a64032f84fadb05f4455c2c51529. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.606245 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 4af0d07a878c42f5a15ff9b8a2295f30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:47.606621 751 tablet_service.cc:1467] Processing CreateTablet for tablet 436eecfe1f34455ead6bad2b884a7cee (DEFAULT_TABLE table=default.Table_ee86b03e71a1459f8fbf833684f9b321 [id=25e4d7a8a23e4603af050b5b1a532375]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.606987 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 436eecfe1f34455ead6bad2b884a7cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.565734 3109 rpcz_store.cc:276] 0112 02:52:45.247818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.247874 (+ 56us) service_pool.cc:225] Handling call
0112 02:52:47.565536 (+2317662us) ts_tablet_manager.cc:1145] Deleting tablet 3d662e0ce13f447595e5f92b94de3018
0112 02:52:47.565545 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.565594 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2317493}
I20250112 02:52:47.607321 3109 tablet_replica.cc:331] T 26577b614e32458e8bc5d6830c8c2e40 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:47.607604 3109 raft_consensus.cc:2238] T 26577b614e32458e8bc5d6830c8c2e40 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:47.607664 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc2b984178a446d193a9dec4dafc64e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.607856 3109 raft_consensus.cc:2267] T 26577b614e32458e8bc5d6830c8c2e40 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:47.608917 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_7048a3b184fc4f85be97bc0227107539" } new_table_name: "default.Table_17388940cd444538a523e34ad3cf51f1" modify_external_catalogs: true
I20250112 02:52:47.610168 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_1a28b756f7994271b098c5404bd23e7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.610603 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a28b756f7994271b098c5404bd23e7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.612010 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 436eecfe1f34455ead6bad2b884a7cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.616830 1173 tablet_bootstrap.cc:492] T ea33053c62dc4707b759d5c1756ab4bd P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:47.616950 1173 ts_tablet_manager.cc:1397] T ea33053c62dc4707b759d5c1756ab4bd P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.286s user 0.003s sys 0.003s
W20250112 02:52:47.621223 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ff7a7cc250c446db0985ffc1bcc2f91 has no consensus state
I20250112 02:52:47.607143 451 catalog_manager.cc:5526] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:47.623234 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_d9117b8cf65240dab1d9c1f61fe05192 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 2 got 0
I20250112 02:52:47.625126 3307 tablet_copy_client.cc:323] T b7340a7a916c4b2fab32e0ba2d0ce322 P f1e775296eb746db8b318f17d26820e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.104.1:35007
I20250112 02:52:47.627506 3273 raft_consensus.cc:491] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:47.627650 3273 raft_consensus.cc:513] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:47.628333 3273 leader_election.cc:290] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:47.630277 3310 tablet_copy_client.cc:323] T 5804fc4130f84c69832c1022b7bd7fa5 P f1e775296eb746db8b318f17d26820e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.104.1:35007
I20250112 02:52:47.630174 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_fc24227c8664449580376c46474e64d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.630659 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc24227c8664449580376c46474e64d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.634809 3273 raft_consensus.cc:491] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:47.634917 3273 raft_consensus.cc:513] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:47.635392 3273 leader_election.cc:290] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:47.637820 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 80080us
W20250112 02:52:47.638332 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 59476.1us
W20250112 02:52:47.638518 989 leader_election.cc:343] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 781a9943f705477cb6eae16df4468d2f
I20250112 02:52:47.638697 989 leader_election.cc:304] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:47.639194 2745 raft_consensus.cc:2798] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:47.639278 2745 raft_consensus.cc:491] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:47.639343 2745 raft_consensus.cc:3054] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:47.641615 2745 raft_consensus.cc:513] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:47.642125 2745 leader_election.cc:290] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:47.644234 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe57d374eb44c3fb1532484e214b654" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:47.644747 1173 raft_consensus.cc:357] T ea33053c62dc4707b759d5c1756ab4bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:47.645177 1173 raft_consensus.cc:383] T ea33053c62dc4707b759d5c1756ab4bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:47.645184 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2796289c7e8f4b45b26508c50b57c640" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:47.645355 1173 raft_consensus.cc:738] T ea33053c62dc4707b759d5c1756ab4bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:47.646544 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 08c1a4ccdb664c05a9999a9843ded765 (DEFAULT_TABLE table=default.Table_36b06d2edb8e4d23a3cb7021a8267db4 [id=3506f201d2674e38a353c87e1b2f26a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.645730 1173 consensus_queue.cc:260] T ea33053c62dc4707b759d5c1756ab4bd P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:47.646623 1027 tablet_service.cc:1467] Processing CreateTablet for tablet a94cec0f0d4d4f06aa5667135efbf5a6 (DEFAULT_TABLE table=default.Table_d3eb6d686d9540ffbdc854d3932aa893 [id=d1fa33e075d74f1193fe708f5034fbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.646821 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 500f0c5c76e54cd8808e6c8611e99d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:47.647254 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 781fe3514f4c4f80b1ee49278502ccc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:47.647404 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet f950c604609042da96e17398f42c1929 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:47.647444 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb8e9a6c6ae4fc9a4b2c0d165c87d91 (DEFAULT_TABLE table=default.Table_4e1bfc47f727446ba5be2309a37017fb [id=b81a3514830f4fb0a458f69803af3200]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.646338 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 1c48318e40954706b2c14a4573b5f5b8 (DEFAULT_TABLE table=default.Table_a1e87ce18198488dac5e9cbbb10fd42e [id=1e6e1e6b5db548c49c3872677fcae8be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.645762 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4a41baf3ad44ffb9be6cd8c526bbcd (DEFAULT_TABLE table=default.Table_2fb335949dc24790a343c60f455a560a [id=c2b1663a1d674d8f8e5d37eeb58fb86c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.648160 1173 ts_tablet_manager.cc:1428] T ea33053c62dc4707b759d5c1756ab4bd P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
I20250112 02:52:47.649758 1173 tablet_bootstrap.cc:492] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:47.646072 1025 tablet_service.cc:1467] Processing CreateTablet for tablet fc2b984178a446d193a9dec4dafc64e8 (DEFAULT_TABLE table=default.Table_b7da5ba607bd47e89d807929a64e2ea7 [id=8a0707ea3b06473d95af04bcd7384ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.652050 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_aef01bf228ad41138e2ffe97ce0e906a" } modify_external_catalogs: true
I20250112 02:52:47.652217 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_aef01bf228ad41138e2ffe97ce0e906a" } modify_external_catalogs: true
I20250112 02:52:47.655822 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_c2314b98707643e39774d4255163325d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.656289 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2314b98707643e39774d4255163325d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.662257 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 37862023095644bab2403576e8cb8c58
I20250112 02:52:47.669250 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_b0d877b624604ddc93c56bec6e05261f" } modify_external_catalogs: true
I20250112 02:52:47.669440 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_b0d877b624604ddc93c56bec6e05261f" } modify_external_catalogs: true
I20250112 02:52:47.672399 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_e071703f861849acab78065ef7c67728"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.672902 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e071703f861849acab78065ef7c67728 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.674090 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_ebadf77c864c4883a04b9f62f15d7f49" } new_table_name: "default.Table_14d8cd331308428f9ce2fff4e64f6394" modify_external_catalogs: true
I20250112 02:52:47.675777 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_6a2dccba50e94d42b01b4ad9e1ca5a35" } modify_external_catalogs: true
I20250112 02:52:47.675949 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_6a2dccba50e94d42b01b4ad9e1ca5a35" } modify_external_catalogs: true
I20250112 02:52:47.677364 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_fe7d42a7f9a645048b94a8eb99cf431b" } new_table_name: "default.Table_72fbcfa736db454c930df858dc3b4baf" modify_external_catalogs: true
W20250112 02:52:47.688382 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 50206.2us
I20250112 02:52:47.642532 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_87c7c83e6cef4175a123fa9924acd6d7" } modify_external_catalogs: true
I20250112 02:52:47.689370 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_87c7c83e6cef4175a123fa9924acd6d7" } modify_external_catalogs: true
I20250112 02:52:47.697762 3332 tablet.cc:1721] T 42444e99aba84a7b873bc8cdf954a6dc P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:47.701246 3245 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca6f080e5fd84084b601b646d4e6c9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.703599 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a94cec0f0d4d4f06aa5667135efbf5a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.703779 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9eb8e9a6c6ae4fc9a4b2c0d165c87d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.707298 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc2b984178a446d193a9dec4dafc64e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.707487 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08c1a4ccdb664c05a9999a9843ded765. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.709730 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bf8a907f0af4e24819d1f00e40885ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:47.711548 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c48318e40954706b2c14a4573b5f5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.711767 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_556e3fb3ccfb4128b7224a024ad819ac" } new_table_name: "default.Table_f957cb19a384456eb663c27c470cf702" modify_external_catalogs: true
I20250112 02:52:47.715579 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4a41baf3ad44ffb9be6cd8c526bbcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.715785 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef34ffff8f24f819d7f1cfafdc639de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.719907 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_bbb2a171a9e346a195035ddcec0eb8fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.720448 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbb2a171a9e346a195035ddcec0eb8fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.720865 1173 tablet_bootstrap.cc:654] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:47.724749 3245 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.104.2:34491 (request call id 460) took 1037 ms. Trace:
I20250112 02:52:47.724885 3245 rpcz_store.cc:276] 0112 02:52:46.687040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.687472 (+ 432us) service_pool.cc:225] Handling call
0112 02:52:46.687731 (+ 259us) ts_tablet_manager.cc:928] T ca6f080e5fd84084b601b646d4e6c9e2 P f1e775296eb746db8b318f17d26820e6: Initiating tablet copy from peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
0112 02:52:47.642724 (+954993us) tablet_copy_client.cc:368] Tablet copy session begun
0112 02:52:47.710536 (+ 67812us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:47.710599 (+ 63us) tablet_copy_client.cc:497] Wrote new tablet metadata
0112 02:52:47.724521 (+ 13922us) tablet_copy_client.cc:418] Wrote new consensus metadata
0112 02:52:47.724617 (+ 96us) tablet_replica.cc:206] Creating consensus instance
0112 02:52:47.724686 (+ 69us) ts_tablet_manager.cc:983] Replica registered.
0112 02:52:47.724704 (+ 18us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":79,"tablet-copy.queue_time_us":30,"thread_start_us":107533,"threads_started":1}
I20250112 02:52:47.726344 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_211137fe64404ad5ba736c0aea830162"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.726800 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_211137fe64404ad5ba736c0aea830162 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.637523 1181 tablet_bootstrap.cc:492] T ca2165362ee541adac724b63f39b1489 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:47.737773 1181 ts_tablet_manager.cc:1397] T ca2165362ee541adac724b63f39b1489 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.496s user 0.008s sys 0.003s
I20250112 02:52:47.658385 3000 raft_consensus.cc:491] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:47.738979 3000 raft_consensus.cc:513] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:47.739873 3000 leader_election.cc:290] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:47.650530 854 leader_election.cc:343] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:47.740212 854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 89711.4us
W20250112 02:52:47.740928 854 leader_election.cc:343] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:47.741688 3000 raft_consensus.cc:491] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:47.741858 3000 raft_consensus.cc:513] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:47.742555 3000 leader_election.cc:290] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:47.743672 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 9ded8bcdda914ea6bc4f9bb92d413dee
I20250112 02:52:47.746814 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ded8bcdda914ea6bc4f9bb92d413dee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:47.664299 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 37862023095644bab2403576e8cb8c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:47.658684 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d418cda9ed5449fa45690130cd6d5bc" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:47.748616 915 raft_consensus.cc:3054] T 3d418cda9ed5449fa45690130cd6d5bc P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:47.752454 3000 raft_consensus.cc:491] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:47.752579 3000 raft_consensus.cc:513] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:47.753216 3000 leader_election.cc:290] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:47.758467 3245 tablet_copy_client.cc:806] T ca6f080e5fd84084b601b646d4e6c9e2 P f1e775296eb746db8b318f17d26820e6: tablet copy: Starting download of 0 data blocks...
I20250112 02:52:47.758716 3245 tablet_copy_client.cc:670] T ca6f080e5fd84084b601b646d4e6c9e2 P f1e775296eb746db8b318f17d26820e6: tablet copy: Starting download of 1 WAL segments...
I20250112 02:52:47.777276 3282 tablet_bootstrap.cc:492] T 164f76edcf8f4bd498dcd2c25fd7477d P f1e775296eb746db8b318f17d26820e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 02:52:47.777700 3282 tablet_bootstrap.cc:492] T 164f76edcf8f4bd498dcd2c25fd7477d P f1e775296eb746db8b318f17d26820e6: Bootstrap complete.
I20250112 02:52:47.777947 3282 ts_tablet_manager.cc:1397] T 164f76edcf8f4bd498dcd2c25fd7477d P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.712s user 0.004s sys 0.004s
I20250112 02:52:47.778752 3282 raft_consensus.cc:357] T 164f76edcf8f4bd498dcd2c25fd7477d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:47.785323 1173 tablet_bootstrap.cc:492] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:47.785449 1173 ts_tablet_manager.cc:1397] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.136s user 0.003s sys 0.003s
I20250112 02:52:47.786067 1173 raft_consensus.cc:357] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:47.786244 1173 raft_consensus.cc:383] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:47.786294 1173 raft_consensus.cc:738] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:47.786477 1173 consensus_queue.cc:260] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:47.786846 1173 ts_tablet_manager.cc:1428] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 02:52:47.787997 744 tablet_service.cc:1514] Processing DeleteTablet for tablet af689c1a3e2d41beb2199bf5d033aa29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:47.788201 744 tablet_service.cc:1514] Processing DeleteTablet for tablet bf2df39addb14d679b8cb7960a62d603 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:47.788341 744 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b64e4e64cc462891f1ee7a594d5026 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:47.561811 1177 raft_consensus.cc:383] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:47.789906 1177 raft_consensus.cc:738] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:47.790104 1177 consensus_queue.cc:260] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:47.790961 3109 ts_tablet_manager.cc:1905] T 26577b614e32458e8bc5d6830c8c2e40 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:47.796355 749 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4a41baf3ad44ffb9be6cd8c526bbcd (DEFAULT_TABLE table=default.Table_2fb335949dc24790a343c60f455a560a [id=c2b1663a1d674d8f8e5d37eeb58fb86c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.796710 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe57d374eb44c3fb1532484e214b654" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:47.798180 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "781a9943f705477cb6eae16df4468d2f" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:47.799985 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ded8bcdda914ea6bc4f9bb92d413dee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:47.800545 754 tablet_service.cc:1467] Processing CreateTablet for tablet fc2b984178a446d193a9dec4dafc64e8 (DEFAULT_TABLE table=default.Table_b7da5ba607bd47e89d807929a64e2ea7 [id=8a0707ea3b06473d95af04bcd7384ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.800966 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc2b984178a446d193a9dec4dafc64e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.803750 760 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:49624 (request call id 519) took 1026 ms. Trace:
I20250112 02:52:47.803833 760 rpcz_store.cc:276] 0112 02:52:46.777429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.778042 (+ 613us) service_pool.cc:225] Handling call
0112 02:52:47.071433 (+293391us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 02:52:47.071484 (+ 51us) ts_tablet_manager.cc:676] Creating new metadata...
0112 02:52:47.414202 (+342718us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:47.803614 (+389412us) tablet_replica.cc:206] Creating consensus instance
0112 02:52:47.803723 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:52:47.804195 760 tablet_service.cc:1467] Processing CreateTablet for tablet 08c1a4ccdb664c05a9999a9843ded765 (DEFAULT_TABLE table=default.Table_36b06d2edb8e4d23a3cb7021a8267db4 [id=3506f201d2674e38a353c87e1b2f26a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.804463 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08c1a4ccdb664c05a9999a9843ded765. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:47.805191 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a6b9a0fc1036472f968da8140b0e315c has no consensus state
I20250112 02:52:47.806041 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 500f0c5c76e54cd8808e6c8611e99d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:47.806167 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 1bd1fb8d69bd411bb02fdafa6ba720f5
I20250112 02:52:47.806299 745 tablet_service.cc:1467] Processing CreateTablet for tablet 1c48318e40954706b2c14a4573b5f5b8 (DEFAULT_TABLE table=default.Table_a1e87ce18198488dac5e9cbbb10fd42e [id=1e6e1e6b5db548c49c3872677fcae8be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.806548 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c48318e40954706b2c14a4573b5f5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.808979 757 tablet_service.cc:1467] Processing CreateTablet for tablet a94cec0f0d4d4f06aa5667135efbf5a6 (DEFAULT_TABLE table=default.Table_d3eb6d686d9540ffbdc854d3932aa893 [id=d1fa33e075d74f1193fe708f5034fbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.809283 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a94cec0f0d4d4f06aa5667135efbf5a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.810299 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_54e45ab6e6d14005bfe1cc06501f6d9a" } modify_external_catalogs: true
I20250112 02:52:47.810473 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_54e45ab6e6d14005bfe1cc06501f6d9a" } modify_external_catalogs: true
I20250112 02:52:47.812002 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_6835dfff9f1f49c898347a250e5f0113"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.812531 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6835dfff9f1f49c898347a250e5f0113 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.813755 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_3140af67d0f7472ea77365c505900416"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:47.814172 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d418cda9ed5449fa45690130cd6d5bc" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:47.814389 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3140af67d0f7472ea77365c505900416 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.814792 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4a41baf3ad44ffb9be6cd8c526bbcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.814937 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 781fe3514f4c4f80b1ee49278502ccc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:47.815074 750 tablet_service.cc:1514] Processing DeleteTablet for tablet f950c604609042da96e17398f42c1929 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:47.814787 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_9a6eb566a9f24077bb60c1c5434165e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.816156 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a6eb566a9f24077bb60c1c5434165e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.817420 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_745092ddecfa47d3a2772ff78e6eab05" } new_table_name: "default.Table_5e1613f069fd44f7ad8f7b54c7d29b72" modify_external_catalogs: true
I20250112 02:52:47.818111 3000 raft_consensus.cc:491] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:47.818284 3000 raft_consensus.cc:513] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:47.819052 3000 leader_election.cc:290] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:47.821162 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bd1fb8d69bd411bb02fdafa6ba720f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:47.823156 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bd1fb8d69bd411bb02fdafa6ba720f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:38786
W20250112 02:52:47.823460 855 leader_election.cc:343] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:47.823606 855 leader_election.cc:304] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:47.823982 3000 raft_consensus.cc:2743] T 8fe57d374eb44c3fb1532484e214b654 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:47.826941 3000 raft_consensus.cc:491] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:47.827066 3000 raft_consensus.cc:513] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:47.827698 3000 leader_election.cc:290] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:47.830004 3000 raft_consensus.cc:491] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:47.830122 3000 raft_consensus.cc:513] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:47.830605 3000 leader_election.cc:290] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:47.831385 3332 tablet.cc:1721] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:47.778945 3282 raft_consensus.cc:738] T 164f76edcf8f4bd498dcd2c25fd7477d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:47.834290 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5450777104da48f1821575658fd31afd" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:47.742425 3111 ts_tablet_manager.cc:1918] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:47.742600 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475b8ea7d68b44fba7431a2812c13871. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.744344 3158 consensus_queue.cc:1035] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:47.836169 3111 log.cc:1198] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/c29a9ca43e034b43a00207e5847bbee3
I20250112 02:52:47.745711 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfe1a1595014c27bbd350cec169d71a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:47.745936 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61eebc5f22664712af6e74e82809ca87" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:47.754779 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecf730b6985404a8134b497a4b792c8" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:47.836313 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2e2a64032f84fadb05f4455c2c51529. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.837205 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 436eecfe1f34455ead6bad2b884a7cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.837350 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc52271535df47e282bf1625c2e8e041" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:47.837817 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8459f094b6e40ea9b5f688f937a9d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.838304 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0925ba2197bb461190b832d062567040. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.838442 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38455a4120f14388b55d94e7fc525c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.838747 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f58a14e35eb41e591afce5a1159c985. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.838929 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a18c4709d4a845309968abafac18f58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.839071 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e6d77ae6ab422a9e1c9a8e1e13183d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.839339 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af4a0d53030406a968b81ca5989303f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.839563 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c3de545ebcb45f6bb72b97126893b44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.841168 3332 tablet.cc:1721] T 4a57eb366f594f93a488c8250e195014 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:47.847560 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 5ea00cab2f6f44b58d82c02a07513416
I20250112 02:52:47.850888 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ea00cab2f6f44b58d82c02a07513416 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:47.851642 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ea00cab2f6f44b58d82c02a07513416 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:47.853228 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_9d1a5f1a606f4ea886dcf4616f6036d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.853713 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d1a5f1a606f4ea886dcf4616f6036d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.854998 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_bd3acd3ed2f141db9e0640f76305391b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.855410 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd3acd3ed2f141db9e0640f76305391b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.856659 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_65c1669814384f6bb9da3a3b1e982d33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.857133 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65c1669814384f6bb9da3a3b1e982d33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.858274 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_26e7eaf3c8dd4451b39f683c627fbd21" } new_table_name: "default.Table_f7499edd95ff4f6d880e1712c2a1a19d" modify_external_catalogs: true
I20250112 02:52:47.859791 1181 raft_consensus.cc:357] T ca2165362ee541adac724b63f39b1489 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:47.860035 1181 raft_consensus.cc:383] T ca2165362ee541adac724b63f39b1489 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:47.860106 1181 raft_consensus.cc:738] T ca2165362ee541adac724b63f39b1489 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:47.860344 1181 consensus_queue.cc:260] T ca2165362ee541adac724b63f39b1489 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:47.861011 1181 ts_tablet_manager.cc:1428] T ca2165362ee541adac724b63f39b1489 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.123s user 0.000s sys 0.002s
I20250112 02:52:47.861330 1181 tablet_bootstrap.cc:492] T 3dab0466476c4a9aa265271c1abd91a4 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:47.863106 1181 tablet_bootstrap.cc:654] T 3dab0466476c4a9aa265271c1abd91a4 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:47.865027 1173 tablet_bootstrap.cc:492] T 93aba9b4d1dd485cb62c4827c916a1d5 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:47.866484 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 37862023095644bab2403576e8cb8c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:47.867359 3111 ts_tablet_manager.cc:1939] T c29a9ca43e034b43a00207e5847bbee3 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:47.867672 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 440) took 1951 ms. Trace:
I20250112 02:52:47.867749 3111 rpcz_store.cc:276] 0112 02:52:45.915812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.916464 (+ 652us) service_pool.cc:225] Handling call
0112 02:52:47.316518 (+1400054us) ts_tablet_manager.cc:1145] Deleting tablet c29a9ca43e034b43a00207e5847bbee3
0112 02:52:47.316528 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.391206 (+ 74678us) spinlock_profiling.cc:243] Waited 70.8 ms on lock 0x616000128e54. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7d0ed0 00007f2e8cac5463 00007f2e8f569eb7 00007f2e8f59c584 00007f2e8f57d690 00007f2e8f57d41d 0000000000a71ecc 00007f2e7ff74b64 00007f2e7ff7818b 00007f2e7ff78008
0112 02:52:47.739871 (+348665us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:47.742362 (+ 2491us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:47.867657 (+125295us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":162885504,"tablet-delete.queue_time_us":1395088}
I20250112 02:52:47.868202 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet c29a9ca43e034b43a00207e5847bbee3 (table default.Table_9284616ca0cd499cb51df0322db9948f [id=7f36d8ba6d864d32ad84be06ca6682ee]) successfully deleted
I20250112 02:52:47.869541 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 68e480204ee94a44a1fef701ef5541f7
I20250112 02:52:47.870474 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 68e480204ee94a44a1fef701ef5541f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:47.813506 779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2796289c7e8f4b45b26508c50b57c640" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:47.814076 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfe1a1595014c27bbd350cec169d71a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:47.876034 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_9fdbd960fe3a41c59ab9262ef27fcf95" } modify_external_catalogs: true
I20250112 02:52:47.876204 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_9fdbd960fe3a41c59ab9262ef27fcf95" } modify_external_catalogs: true
I20250112 02:52:47.876492 1177 ts_tablet_manager.cc:1428] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.399s user 0.001s sys 0.001s
I20250112 02:52:47.876806 1177 tablet_bootstrap.cc:492] T 83c04f1a2cb04950a9f6170e252c19b4 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:47.858783 3282 consensus_queue.cc:260] T 164f76edcf8f4bd498dcd2c25fd7477d P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:47.881609 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecf730b6985404a8134b497a4b792c8" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:47.881609 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61eebc5f22664712af6e74e82809ca87" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:47.882193 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc52271535df47e282bf1625c2e8e041" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:47.882788 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a470f84d50df4cf0b4a1e5d248b4c32a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:47.883901 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d418cda9ed5449fa45690130cd6d5bc" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:47.885284 800 tablet_copy_service.cc:140] P 289afc116aa74408aa5bff74e11c50d8: Received BeginTabletCopySession request for tablet b7340a7a916c4b2fab32e0ba2d0ce322 from peer f1e775296eb746db8b318f17d26820e6 ({username='slave'} at 127.0.104.3:52505)
I20250112 02:52:47.885356 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 449) took 1886 ms. Trace:
I20250112 02:52:47.885408 800 tablet_copy_service.cc:161] P 289afc116aa74408aa5bff74e11c50d8: Beginning new tablet copy session on tablet b7340a7a916c4b2fab32e0ba2d0ce322 from peer f1e775296eb746db8b318f17d26820e6 at {username='slave'} at 127.0.104.3:52505: session id = f1e775296eb746db8b318f17d26820e6-b7340a7a916c4b2fab32e0ba2d0ce322
I20250112 02:52:47.885452 3111 rpcz_store.cc:276] 0112 02:52:45.999331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.002007 (+ 2676us) service_pool.cc:225] Handling call
0112 02:52:47.870036 (+1868029us) ts_tablet_manager.cc:1145] Deleting tablet f60626faf203441ab48c60147ddc3cf8
0112 02:52:47.870046 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.885333 (+ 15287us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1867820}
W20250112 02:52:47.885900 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f60626faf203441ab48c60147ddc3cf8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f60626faf203441ab48c60147ddc3cf8 already in progress: creating tablet
I20250112 02:52:47.887575 799 tablet_copy_service.cc:140] P 289afc116aa74408aa5bff74e11c50d8: Received BeginTabletCopySession request for tablet 5804fc4130f84c69832c1022b7bd7fa5 from peer f1e775296eb746db8b318f17d26820e6 ({username='slave'} at 127.0.104.3:52505)
I20250112 02:52:47.887688 799 tablet_copy_service.cc:161] P 289afc116aa74408aa5bff74e11c50d8: Beginning new tablet copy session on tablet 5804fc4130f84c69832c1022b7bd7fa5 from peer f1e775296eb746db8b318f17d26820e6 at {username='slave'} at 127.0.104.3:52505: session id = f1e775296eb746db8b318f17d26820e6-5804fc4130f84c69832c1022b7bd7fa5
I20250112 02:52:47.888077 1173 tablet_bootstrap.cc:654] T 93aba9b4d1dd485cb62c4827c916a1d5 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:47.889015 3282 ts_tablet_manager.cc:1428] T 164f76edcf8f4bd498dcd2c25fd7477d P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.111s user 0.004s sys 0.000s
I20250112 02:52:47.890736 799 tablet_copy_source_session.cc:215] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 02:52:47.891863 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5450777104da48f1821575658fd31afd" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:47.895584 800 tablet_copy_source_session.cc:215] T b7340a7a916c4b2fab32e0ba2d0ce322 P 289afc116aa74408aa5bff74e11c50d8: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 02:52:47.896267 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 68e480204ee94a44a1fef701ef5541f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:47.900754 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f810f10151c84e648b338a55fb181277 consensus state has no leader
I20250112 02:52:47.903676 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 451) took 1822 ms. Trace:
I20250112 02:52:47.904129 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_6fce610f7bc74a7d9ec1c48ff2bd5a95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.904695 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6fce610f7bc74a7d9ec1c48ff2bd5a95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:47.910470 855 leader_election.cc:343] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:47.911136 855 leader_election.cc:304] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:47.911845 855 leader_election.cc:343] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:47.899037 768 raft_consensus.cc:3054] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:47.899545 759 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb8e9a6c6ae4fc9a4b2c0d165c87d91 (DEFAULT_TABLE table=default.Table_4e1bfc47f727446ba5be2309a37017fb [id=b81a3514830f4fb0a458f69803af3200]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.915953 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9eb8e9a6c6ae4fc9a4b2c0d165c87d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:47.899951 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af4a0d53030406a968b81ca5989303f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:47.900292 752 consensus_meta.cc:287] T 7d0d84297810420b8b7a4b24841eda55 P 289afc116aa74408aa5bff74e11c50d8: Time spent flushing consensus metadata: real 0.581s user 0.002s sys 0.000s
I20250112 02:52:47.903756 3111 rpcz_store.cc:276] 0112 02:52:46.081023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.081702 (+ 679us) service_pool.cc:225] Handling call
0112 02:52:47.886825 (+1805123us) ts_tablet_manager.cc:1145] Deleting tablet 37d38f5e525440f8a0f7822620933435
0112 02:52:47.886834 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.903659 (+ 16825us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1804967}
W20250112 02:52:47.920153 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 37d38f5e525440f8a0f7822620933435 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37d38f5e525440f8a0f7822620933435 already in progress: creating tablet
I20250112 02:52:47.898540 747 tablet_service.cc:1467] Processing CreateTablet for tablet a2e2a64032f84fadb05f4455c2c51529 (DEFAULT_TABLE table=default.Table_866e2b7327c64126af36d106ef79ce17 [id=d020634d12fe45cfac5dbada38a37a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:47.924060 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2e2a64032f84fadb05f4455c2c51529. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:47.929546 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d9e50a33ac74482aa9d4cdf9d61e91a has no consensus state
I20250112 02:52:47.931573 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 452) took 1850 ms. Trace:
I20250112 02:52:47.931665 3111 rpcz_store.cc:276] 0112 02:52:46.081145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.081882 (+ 737us) service_pool.cc:225] Handling call
0112 02:52:47.919946 (+1838064us) ts_tablet_manager.cc:1145] Deleting tablet cd52261c4a414e7d93a528e2eddfa9db
0112 02:52:47.919955 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.931555 (+ 11600us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1837937}
I20250112 02:52:47.940644 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet d1269a6ed80e46ecb7f4f87e5afb732f
W20250112 02:52:47.938664 3289 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.002s sys 0.027s
W20250112 02:52:47.944180 3289 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s user 0.002s sys 0.027s
W20250112 02:52:47.944993 3144 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.005s sys 0.000s
W20250112 02:52:47.945154 3144 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.591s user 0.005s sys 0.000s
I20250112 02:52:47.947619 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_6f3879f2469d4655a6d4436362d0c3bb" } new_table_name: "default.Table_a6e7b6bac5ba4d1faf8e414b6463a25a" modify_external_catalogs: true
I20250112 02:52:47.949636 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_cc892d530b634f7cad6df1f05af3560e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:47.950230 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc892d530b634f7cad6df1f05af3560e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:47.951848 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_1ccecb4c9e1f481b9c467fea54894390" } modify_external_catalogs: true
I20250112 02:52:47.952083 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_1ccecb4c9e1f481b9c467fea54894390" } modify_external_catalogs: true
W20250112 02:52:47.954567 855 leader_election.cc:343] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:47.957010 1177 tablet_bootstrap.cc:654] T 83c04f1a2cb04950a9f6170e252c19b4 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:47.957440 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet f810f10151c84e648b338a55fb181277 (table default.Table_5e1613f069fd44f7ad8f7b54c7d29b72 [id=b6ffbb39eb124fba8a5c4d048613acfa], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f810f10151c84e648b338a55fb181277 consensus state has no leader
W20250112 02:52:47.958182 713 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.018s sys 0.032s
W20250112 02:52:47.958343 713 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.573s user 0.018s sys 0.032s
W20250112 02:52:47.958485 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 572623us
I20250112 02:52:47.959789 3296 consensus_queue.cc:1035] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:47.929046 3273 raft_consensus.cc:491] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:47.961614 3273 raft_consensus.cc:513] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:47.962188 3273 leader_election.cc:290] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:47.962889 855 leader_election.cc:343] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:47.963052 855 leader_election.cc:343] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:47.963191 855 leader_election.cc:343] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:47.963752 855 consensus_peers.cc:487] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:52:47.963737 3304 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s user 0.011s sys 0.033s
W20250112 02:52:47.963922 855 leader_election.cc:343] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:47.964071 3304 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.639s user 0.012s sys 0.033s
I20250112 02:52:47.912030 3000 raft_consensus.cc:2743] T 2796289c7e8f4b45b26508c50b57c640 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:47.964926 3352 tablet.cc:1721] T 7cd9985cd20148f195a857c478b0755f P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 02:52:47.963755 772 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.015s sys 0.028s
W20250112 02:52:47.966348 772 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s user 0.015s sys 0.028s
I20250112 02:52:47.698369 3335 tablet.cc:1721] T 42444e99aba84a7b873bc8cdf954a6dc P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:47.973367 3335 tablet.cc:1721] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:47.976032 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet d1269a6ed80e46ecb7f4f87e5afb732f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:47.978083 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 450) took 1897 ms. Trace:
I20250112 02:52:47.978178 3111 rpcz_store.cc:276] 0112 02:52:46.080943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.082399 (+ 1456us) service_pool.cc:225] Handling call
0112 02:52:47.931977 (+1849578us) ts_tablet_manager.cc:1145] Deleting tablet d2028226498b4b41a95c7c836fe03b4d
0112 02:52:47.931985 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.978059 (+ 46074us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1849328}
W20250112 02:52:47.978569 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet d2028226498b4b41a95c7c836fe03b4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2028226498b4b41a95c7c836fe03b4d already in progress: creating tablet
I20250112 02:52:47.979563 3352 tablet.cc:1721] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:47.985419 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 457) took 1602 ms. Trace:
I20250112 02:52:47.985493 3111 rpcz_store.cc:276] 0112 02:52:46.382592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.382635 (+ 43us) service_pool.cc:225] Handling call
0112 02:52:47.978574 (+1595939us) ts_tablet_manager.cc:1145] Deleting tablet 5804fc4130f84c69832c1022b7bd7fa5
0112 02:52:47.978603 (+ 29us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.985400 (+ 6797us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1594097}
I20250112 02:52:47.985865 3111 tablet_replica.cc:331] T 3499374810fb412e938325434c35618d P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:47.986047 3111 raft_consensus.cc:2238] T 3499374810fb412e938325434c35618d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:47.986207 3111 raft_consensus.cc:2267] T 3499374810fb412e938325434c35618d P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:52:47.986478 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 5804fc4130f84c69832c1022b7bd7fa5 because the tablet was not found. No further retry: Not found: Tablet not found: 5804fc4130f84c69832c1022b7bd7fa5
I20250112 02:52:47.990994 3111 ts_tablet_manager.cc:1905] T 3499374810fb412e938325434c35618d P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:47.991230 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet a18c4709d4a845309968abafac18f58f
I20250112 02:52:47.994482 3158 raft_consensus.cc:491] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:47.994621 3158 raft_consensus.cc:513] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:47.993837 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet a18c4709d4a845309968abafac18f58f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:47.995282 3158 leader_election.cc:290] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:47.996052 1173 tablet_bootstrap.cc:492] T 93aba9b4d1dd485cb62c4827c916a1d5 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:47.996155 1173 ts_tablet_manager.cc:1397] T 93aba9b4d1dd485cb62c4827c916a1d5 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.131s user 0.005s sys 0.005s
I20250112 02:52:48.000092 1173 raft_consensus.cc:357] T 93aba9b4d1dd485cb62c4827c916a1d5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.000252 1173 raft_consensus.cc:383] T 93aba9b4d1dd485cb62c4827c916a1d5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:48.000294 1173 raft_consensus.cc:738] T 93aba9b4d1dd485cb62c4827c916a1d5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:48.000464 1173 consensus_queue.cc:260] T 93aba9b4d1dd485cb62c4827c916a1d5 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.005456 1173 ts_tablet_manager.cc:1428] T 93aba9b4d1dd485cb62c4827c916a1d5 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250112 02:52:48.005685 1173 tablet_bootstrap.cc:492] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:48.007061 1173 tablet_bootstrap.cc:654] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:48.011698 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_c7dcf1defec042dabf093c4b9b0449a7" } modify_external_catalogs: true
I20250112 02:52:48.011883 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_c7dcf1defec042dabf093c4b9b0449a7" } modify_external_catalogs: true
I20250112 02:52:48.011422 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_178713b464bb4f16970851646a710574" } new_table_name: "default.Table_3cd1b394e24e4ee7860e55095a74fffd" modify_external_catalogs: true
W20250112 02:52:47.976665 3301 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s user 0.014s sys 0.029s
W20250112 02:52:48.015988 3301 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.661s user 0.014s sys 0.029s
I20250112 02:52:48.012138 3133 consensus_queue.cc:1035] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250112 02:52:48.019183 3111 ts_tablet_manager.cc:1918] T 3499374810fb412e938325434c35618d P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:48.019327 3111 log.cc:1198] T 3499374810fb412e938325434c35618d P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/3499374810fb412e938325434c35618d
W20250112 02:52:48.021170 3349 log.cc:926] Time spent T 79eb61cedd544c49869471eebe2fd237 P 289afc116aa74408aa5bff74e11c50d8: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
W20250112 02:52:48.022342 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 818cd6362b94473ea06ed68fa5ffd1bc consensus state has no leader
I20250112 02:52:48.023121 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 38e4a69f470a4d62b865e2dd594fab63
I20250112 02:52:48.028298 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 38e4a69f470a4d62b865e2dd594fab63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:48.028504 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 4af0d07a878c42f5a15ff9b8a2295f30 (DEFAULT_TABLE table=default.Table_b910dc09eda7442ea8f6df224e60e7db [id=1b55644d4bb14110a8c37d9a0dd2eccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.030097 1035 tablet_service.cc:1467] Processing CreateTablet for tablet ad7b2e22c9584f55b5a4c996bd305e13 (DEFAULT_TABLE table=default.Table_518ec5f7946342419e87f093585388fe [id=55b32a6c95f94c2a8bf3f216238cdc92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.031947 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4af0d07a878c42f5a15ff9b8a2295f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.032061 3111 ts_tablet_manager.cc:1939] T 3499374810fb412e938325434c35618d P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:48.032344 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 458) took 1649 ms. Trace:
I20250112 02:52:48.032406 3111 rpcz_store.cc:276] 0112 02:52:46.382698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.382751 (+ 53us) service_pool.cc:225] Handling call
0112 02:52:47.985822 (+1603071us) ts_tablet_manager.cc:1145] Deleting tablet 3499374810fb412e938325434c35618d
0112 02:52:47.985830 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.010983 (+ 25153us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.019051 (+ 8068us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.032329 (+ 13278us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1601274}
I20250112 02:52:48.032681 3111 tablet_replica.cc:331] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:48.032836 3111 raft_consensus.cc:2238] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:48.033118 3111 raft_consensus.cc:2267] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:48.034176 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad7b2e22c9584f55b5a4c996bd305e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.035490 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 3499374810fb412e938325434c35618d (table default.Table_7870d93893b74b4b96d1a3809a9015e6 [id=4355589b5ccc4162ab8700e31173d1e5]) successfully deleted
I20250112 02:52:48.044859 3365 tablet.cc:1721] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.050341 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_4194f66a2f28431c89e268ef7df1bb66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:48.050989 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4194f66a2f28431c89e268ef7df1bb66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.051725 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_57b3861e9f704dc09d466241c1926fc1" } modify_external_catalogs: true
I20250112 02:52:48.053889 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_57b3861e9f704dc09d466241c1926fc1" } modify_external_catalogs: true
I20250112 02:52:48.054875 3111 ts_tablet_manager.cc:1905] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:48.060310 3111 ts_tablet_manager.cc:1918] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:48.060433 3111 log.cc:1198] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/b3c5a861a90840d0a33b780ec75bffd2
I20250112 02:52:48.062363 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_bdcd9ae330ee40d7881d369688794d8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:48.063529 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdcd9ae330ee40d7881d369688794d8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:48.064477 991 leader_election.cc:343] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.064595 991 leader_election.cc:304] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:48.064885 2745 raft_consensus.cc:2743] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:48.065049 991 leader_election.cc:343] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.065392 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_21743df183e844c9a6ca2031b2bc81bc" } new_table_name: "default.Table_21d1a0a569e34a20a180e2c21f0e6a38" modify_external_catalogs: true
W20250112 02:52:48.065605 991 leader_election.cc:343] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:48.065791 991 consensus_peers.cc:487] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:48.066588 3159 consensus_queue.cc:1035] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:48.050882 3335 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:38786 (request call id 416) took 1189 ms. Trace:
I20250112 02:52:48.072489 447 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 54b1182029af4678974df2a48c2a0dfd
I20250112 02:52:48.076135 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_24fca5290fd54774bee413c9d84a997d" } modify_external_catalogs: true
I20250112 02:52:48.076368 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_24fca5290fd54774bee413c9d84a997d" } modify_external_catalogs: true
I20250112 02:52:48.078152 3307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7340a7a916c4b2fab32e0ba2d0ce322. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.078404 3310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5804fc4130f84c69832c1022b7bd7fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.083099 3352 tablet.cc:1721] T 05000c6a48e0450d8e6685ca864f67f0 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.083299 3348 tablet.cc:1721] T 2052b4536f9745f0a785f330d706c1cf P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:52:48.086073 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a66705bc87074cc8a661f351a6045e99 consensus state has no leader
I20250112 02:52:48.072619 3335 rpcz_store.cc:276] 0112 02:52:46.861681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.867208 (+ 5527us) service_pool.cc:225] Handling call
0112 02:52:48.050865 (+1183657us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:46.867789 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 02:52:46.867870 (+ 81us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 02:52:46.867879 (+ 9us) alter_schema_op.cc:66] Acquired schema lock
0112 02:52:46.867885 (+ 6us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 02:52:46.867916 (+ 31us) alter_schema_op.cc:128] START. Timestamp: 1736650366867791
0112 02:52:46.867919 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:46.868083 (+ 164us) log.cc:844] Serialized 173 byte log entry
0112 02:52:47.964575 (+1096492us) op_driver.cc:464] REPLICATION: finished
0112 02:52:47.973349 (+ 8774us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 02:52:47.973680 (+ 331us) log.cc:844] Serialized 5 byte log entry
0112 02:52:47.999447 (+ 25767us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.050618 (+ 51171us) log.cc:844] Serialized 14 byte log entry
0112 02:52:48.050822 (+ 204us) alter_schema_op.cc:73] Released schema lock
0112 02:52:48.050831 (+ 9us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":8730,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":32114,"prepare.run_wall_time_us":797201,"replication_time_us":1096617,"thread_start_us":792078,"threads_started":2,"wal-append.queue_time_us":639736}]]}
I20250112 02:52:48.088886 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_4215db3e1bc34b188fdda22e81f7d325"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:48.089540 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4215db3e1bc34b188fdda22e81f7d325 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.091099 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_45898d7699014943b4a0d8d270a219ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:48.091728 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45898d7699014943b4a0d8d270a219ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.092162 3307 tablet_copy_client.cc:806] T b7340a7a916c4b2fab32e0ba2d0ce322 P f1e775296eb746db8b318f17d26820e6: tablet copy: Starting download of 0 data blocks...
I20250112 02:52:48.092344 3307 tablet_copy_client.cc:670] T b7340a7a916c4b2fab32e0ba2d0ce322 P f1e775296eb746db8b318f17d26820e6: tablet copy: Starting download of 1 WAL segments...
I20250112 02:52:48.092723 3111 ts_tablet_manager.cc:1939] T b3c5a861a90840d0a33b780ec75bffd2 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:48.093061 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 459) took 1710 ms. Trace:
I20250112 02:52:48.093143 3111 rpcz_store.cc:276] 0112 02:52:46.382783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.382878 (+ 95us) service_pool.cc:225] Handling call
0112 02:52:48.032646 (+1649768us) ts_tablet_manager.cc:1145] Deleting tablet b3c5a861a90840d0a33b780ec75bffd2
0112 02:52:48.032653 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.057581 (+ 24928us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.060233 (+ 2652us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.093045 (+ 32812us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1646969}
I20250112 02:52:48.093597 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet b3c5a861a90840d0a33b780ec75bffd2 (table default.Table_c26869f6cbd540fe980dcf207b5c729d [id=cdd6a2097792484d84bca856f30d7ac1]) successfully deleted
I20250112 02:52:47.737491 3110 ts_tablet_manager.cc:1918] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:48.102849 3110 log.cc:1198] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/e4d42f1611de49d6b6e5aaa9be907054
I20250112 02:52:48.103261 3110 ts_tablet_manager.cc:1939] T e4d42f1611de49d6b6e5aaa9be907054 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:48.104091 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 363) took 2480 ms. Trace:
I20250112 02:52:48.104192 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 2ecf730b6985404a8134b497a4b792c8
I20250112 02:52:48.105592 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_2181502249a64682b8c3a00be9542f67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:48.106487 3348 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:56348 (request call id 512) took 1065 ms. Trace:
I20250112 02:52:48.107120 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ecf730b6985404a8134b497a4b792c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:48.107410 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 460) took 1724 ms. Trace:
I20250112 02:52:48.108059 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet b1d9fae290ec4281839991f9d40b105c (table default.Table_c1639450ddf94784a40b8cf28f3059c4 [id=93cb911ed4e045c994c9a06b10e2e4a8]) successfully deleted
W20250112 02:52:48.108973 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 93aba9b4d1dd485cb62c4827c916a1d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93aba9b4d1dd485cb62c4827c916a1d5 already in progress: creating tablet
W20250112 02:52:48.109290 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 2361bc6bd8164c2daa37be6b0845f4bd because the tablet was not found. No further retry: Not found: Tablet not found: 2361bc6bd8164c2daa37be6b0845f4bd
W20250112 02:52:48.110822 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2181502249a64682b8c3a00be9542f67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.111640 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_a5b8c2cd15dd47818378fc3a3065b890" } new_table_name: "default.Table_1e8424b3da01469588c376566545694f" modify_external_catalogs: true
I20250112 02:52:48.113524 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_78d5ae66b2c74b1aadf6f11115f258be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:48.113996 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78d5ae66b2c74b1aadf6f11115f258be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:52:48.118181 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b664fabebd844f2ab744755ceb150511 consensus state has no leader
I20250112 02:52:48.119784 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet ea33053c62dc4707b759d5c1756ab4bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:48.122118 1173 tablet_bootstrap.cc:492] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:48.122263 1173 ts_tablet_manager.cc:1397] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.117s user 0.016s sys 0.032s
I20250112 02:52:48.129402 1173 raft_consensus.cc:357] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.129614 1173 raft_consensus.cc:383] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:48.106576 3348 rpcz_store.cc:276] 0112 02:52:47.041340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.058194 (+ 16854us) service_pool.cc:225] Handling call
0112 02:52:48.106474 (+1048280us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:47.062448 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 02:52:47.062557 (+ 109us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 02:52:47.062566 (+ 9us) alter_schema_op.cc:66] Acquired schema lock
0112 02:52:47.062574 (+ 8us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 02:52:47.062607 (+ 33us) alter_schema_op.cc:128] START. Timestamp: 1736650367062539
0112 02:52:47.062621 (+ 14us) op_driver.cc:348] REPLICATION: starting
0112 02:52:47.062766 (+ 145us) log.cc:844] Serialized 173 byte log entry
0112 02:52:48.067038 (+1004272us) op_driver.cc:464] REPLICATION: finished
0112 02:52:48.079226 (+ 12188us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 02:52:48.085724 (+ 6498us) log.cc:844] Serialized 5 byte log entry
0112 02:52:48.105318 (+ 19594us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.106402 (+ 1084us) log.cc:844] Serialized 14 byte log entry
0112 02:52:48.106442 (+ 40us) alter_schema_op.cc:73] Released schema lock
0112 02:52:48.106446 (+ 4us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":12118,"prepare.queue_time_us":3935,"prepare.run_cpu_time_us":6211,"prepare.run_wall_time_us":39861,"raft.queue_time_us":328,"raft.run_cpu_time_us":477,"raft.run_wall_time_us":474,"replication_time_us":1004369,"thread_start_us":37310,"threads_started":1,"wal-append.queue_time_us":36959}]]}
I20250112 02:52:48.132473 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_75a41a4f8be84a56b1103d08e0510d4a" } new_table_name: "default.Table_fdce4935e27640fbb50682206a2c6ebf" modify_external_catalogs: true
I20250112 02:52:48.129668 1173 raft_consensus.cc:738] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:48.132767 1173 consensus_queue.cc:260] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.133931 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_a9249382d49d479bb4af7de4d2273661"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:48.134479 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9249382d49d479bb4af7de4d2273661 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.135777 1173 ts_tablet_manager.cc:1428] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250112 02:52:48.139600 3310 tablet_copy_client.cc:806] T 5804fc4130f84c69832c1022b7bd7fa5 P f1e775296eb746db8b318f17d26820e6: tablet copy: Starting download of 0 data blocks...
I20250112 02:52:48.141436 3310 tablet_copy_client.cc:670] T 5804fc4130f84c69832c1022b7bd7fa5 P f1e775296eb746db8b318f17d26820e6: tablet copy: Starting download of 1 WAL segments...
I20250112 02:52:48.107605 3111 rpcz_store.cc:276] 0112 02:52:46.382889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.382940 (+ 51us) service_pool.cc:225] Handling call
0112 02:52:48.093669 (+1710729us) ts_tablet_manager.cc:1145] Deleting tablet 61d7703b15024fe7a35805345d5e81ab
0112 02:52:48.093680 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.107394 (+ 13714us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1698318}
W20250112 02:52:48.142012 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 61d7703b15024fe7a35805345d5e81ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61d7703b15024fe7a35805345d5e81ab already in progress: creating tablet
I20250112 02:52:48.143107 1173 tablet_bootstrap.cc:492] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:48.145737 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f58a14e35eb41e591afce5a1159c985 consensus state has no leader
I20250112 02:52:48.148600 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_5e6ad2a2040d41b381bb2653c6f691ec" } modify_external_catalogs: true
I20250112 02:52:48.148877 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_5e6ad2a2040d41b381bb2653c6f691ec" } modify_external_catalogs: true
I20250112 02:52:48.149040 3110 rpcz_store.cc:276] 0112 02:52:45.623702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.623856 (+ 154us) service_pool.cc:225] Handling call
0112 02:52:47.267401 (+1643545us) ts_tablet_manager.cc:1145] Deleting tablet e4d42f1611de49d6b6e5aaa9be907054
0112 02:52:47.267410 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:47.334167 (+ 66757us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:47.737392 (+403225us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.104072 (+366680us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1643341}
I20250112 02:52:48.149386 3110 tablet_replica.cc:331] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
W20250112 02:52:47.958266 854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 127097us
I20250112 02:52:48.149863 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a470f84d50df4cf0b4a1e5d248b4c32a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:48.150040 1050 raft_consensus.cc:2463] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe36951a83fb4e57b26e187edbcbfe3d in term 0.
I20250112 02:52:48.151242 3347 consensus_queue.cc:1035] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:48.152630 3348 tablet.cc:1721] T 7cd9985cd20148f195a857c478b0755f P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:47.957129 915 raft_consensus.cc:2463] T 3d418cda9ed5449fa45690130cd6d5bc P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
W20250112 02:52:48.129331 855 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 165216us
I20250112 02:52:48.154949 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet e4d42f1611de49d6b6e5aaa9be907054 (table default.Table_f952a0479e8d478bbd9484fc5bf041c8 [id=c01b653cd9fe45b2af721eb97936c5d4]) successfully deleted
I20250112 02:52:48.155998 893 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb8e9a6c6ae4fc9a4b2c0d165c87d91 (DEFAULT_TABLE table=default.Table_4e1bfc47f727446ba5be2309a37017fb [id=b81a3514830f4fb0a458f69803af3200]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.156361 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9eb8e9a6c6ae4fc9a4b2c0d165c87d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.158358 897 tablet_service.cc:1514] Processing DeleteTablet for tablet a18c4709d4a845309968abafac18f58f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:48.158324 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_30dbb158d46847cb903950fdf3e3f1aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:48.158944 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30dbb158d46847cb903950fdf3e3f1aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.166227 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 38e4a69f470a4d62b865e2dd594fab63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:48.169652 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb7e0181cf74bbea6036c106516a7b3" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:48.169756 3356 log.cc:926] Time spent T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20250112 02:52:48.170792 1177 tablet_bootstrap.cc:492] T 83c04f1a2cb04950a9f6170e252c19b4 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:48.170974 1177 ts_tablet_manager.cc:1397] T 83c04f1a2cb04950a9f6170e252c19b4 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.294s user 0.010s sys 0.018s
I20250112 02:52:48.171787 1177 raft_consensus.cc:357] T 83c04f1a2cb04950a9f6170e252c19b4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.172016 1177 raft_consensus.cc:383] T 83c04f1a2cb04950a9f6170e252c19b4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:48.172111 1177 raft_consensus.cc:738] T 83c04f1a2cb04950a9f6170e252c19b4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:48.172346 1177 consensus_queue.cc:260] T 83c04f1a2cb04950a9f6170e252c19b4 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.172806 1177 ts_tablet_manager.cc:1428] T 83c04f1a2cb04950a9f6170e252c19b4 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 02:52:48.173074 1177 tablet_bootstrap.cc:492] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:48.182209 454 master_service.cc:946] ReplaceTablet: received request to replace tablet acafc4f579534a69a8f307ca4cf72e12 from {username='slave'} at 127.0.0.1:40610
W20250112 02:52:48.169773 714 consensus_peers.cc:487] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8 -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05000c6a48e0450d8e6685ca864f67f0. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:48.189361 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_ac3a92da0f3e4df19577758643bfb95d" } new_table_name: "default.Table_b529da05fc294c69bca4cfa030291c41" modify_external_catalogs: true
I20250112 02:52:48.189447 893 tablet_service.cc:1467] Processing CreateTablet for tablet 4af0d07a878c42f5a15ff9b8a2295f30 (DEFAULT_TABLE table=default.Table_b910dc09eda7442ea8f6df224e60e7db [id=1b55644d4bb14110a8c37d9a0dd2eccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.189491 3335 tablet.cc:1721] T ead501d76bb1412e9839fb36b9911e6f P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:48.190292 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4af0d07a878c42f5a15ff9b8a2295f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.190661 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_d9871dbfe3f84ae88ade407167ea927e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:48.191085 3332 tablet.cc:1721] T 46d1beac4410422bbf1e912d35999f3d P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 02:52:48.191218 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9871dbfe3f84ae88ade407167ea927e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.193477 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_15bf41d2cd254fbc8ec6477f1b3deb55" } modify_external_catalogs: true
I20250112 02:52:48.193694 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_15bf41d2cd254fbc8ec6477f1b3deb55" } modify_external_catalogs: true
I20250112 02:52:48.195174 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ecf730b6985404a8134b497a4b792c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:48.195145 3352 tablet.cc:1721] T 7320ac55dbfa47fe851fff94d32e1ffe P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.195467 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 26577b614e32458e8bc5d6830c8c2e40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:48.195550 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 61eac6b572ff47c185768bee7490edf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:45 UTC) from {username='slave'} at 127.0.0.1:38786
W20250112 02:52:48.195807 855 leader_election.cc:343] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.193744 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_bce57ffa52f944bb8e51c0cfb13c497d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:48.197221 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bce57ffa52f944bb8e51c0cfb13c497d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.197616 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 164f76edcf8f4bd498dcd2c25fd7477d
I20250112 02:52:48.195914 885 tablet_service.cc:1467] Processing CreateTablet for tablet 81bad885cbbf4a8b9a06c8a066f25e4c (DEFAULT_TABLE table=default.Table_a6857eb18f404b15a97dade22ad383c1 [id=0f032dab356b4f1fb540c52e22153f99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.202925 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81bad885cbbf4a8b9a06c8a066f25e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.206259 3266 consensus_queue.cc:1035] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:48.196226 888 tablet_service.cc:1467] Processing CreateTablet for tablet e438e49f1e5345a78e11b66ac8e95d28 (DEFAULT_TABLE table=default.Table_80b349ce76644bcabf626e044231436a [id=ba3ca203a209474184c7763dabea3f5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.211136 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 164f76edcf8f4bd498dcd2c25fd7477d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:48.211256 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e438e49f1e5345a78e11b66ac8e95d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.196875 3355 tablet.cc:1721] T 79eb61cedd544c49869471eebe2fd237 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:48.216250 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 164f76edcf8f4bd498dcd2c25fd7477d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:48.197343 886 tablet_service.cc:1467] Processing CreateTablet for tablet ad7b2e22c9584f55b5a4c996bd305e13 (DEFAULT_TABLE table=default.Table_518ec5f7946342419e87f093585388fe [id=55b32a6c95f94c2a8bf3f216238cdc92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.217006 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad7b2e22c9584f55b5a4c996bd305e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.189132 3109 ts_tablet_manager.cc:1918] T 26577b614e32458e8bc5d6830c8c2e40 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:48.201311 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a470f84d50df4cf0b4a1e5d248b4c32a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:48.222589 1021 tablet_service.cc:1467] Processing CreateTablet for tablet f5492416a0544951bcbf8a4f84942e66 (DEFAULT_TABLE table=default.Table_f8ab3098ff7f4c479588888eac14e9a7 [id=a1bcfb4b28f54654ac905844df2ef79d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.227723 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 462) took 1669 ms. Trace:
I20250112 02:52:48.227813 3111 rpcz_store.cc:276] 0112 02:52:46.558251 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.558282 (+ 31us) service_pool.cc:225] Handling call
0112 02:52:48.142024 (+1583742us) ts_tablet_manager.cc:1145] Deleting tablet a1123077650c4e89bffddd6d1264e428
0112 02:52:48.152457 (+ 10433us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.227707 (+ 75250us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1583609}
I20250112 02:52:48.222962 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 81bad885cbbf4a8b9a06c8a066f25e4c (DEFAULT_TABLE table=default.Table_a6857eb18f404b15a97dade22ad383c1 [id=0f032dab356b4f1fb540c52e22153f99]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:48.230782 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet a1123077650c4e89bffddd6d1264e428 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1123077650c4e89bffddd6d1264e428 already in progress: creating tablet
I20250112 02:52:48.231206 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5492416a0544951bcbf8a4f84942e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.231578 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb7e0181cf74bbea6036c106516a7b3" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:48.231859 1035 tablet_service.cc:1467] Processing CreateTablet for tablet e438e49f1e5345a78e11b66ac8e95d28 (DEFAULT_TABLE table=default.Table_80b349ce76644bcabf626e044231436a [id=ba3ca203a209474184c7763dabea3f5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.232235 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e438e49f1e5345a78e11b66ac8e95d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.232739 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81bad885cbbf4a8b9a06c8a066f25e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.233505 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 463) took 1663 ms. Trace:
I20250112 02:52:48.233572 3111 rpcz_store.cc:276] 0112 02:52:46.569955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.570332 (+ 377us) service_pool.cc:225] Handling call
0112 02:52:48.228057 (+1657725us) ts_tablet_manager.cc:1145] Deleting tablet f5c671e327a148ecb1d08a983448946e
0112 02:52:48.233391 (+ 5334us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.233490 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1656042}
I20250112 02:52:48.233803 3111 tablet_replica.cc:331] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
W20250112 02:52:48.235618 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet f5c671e327a148ecb1d08a983448946e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5c671e327a148ecb1d08a983448946e already in progress: creating tablet
I20250112 02:52:48.245142 1013 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50840 (request call id 134) took 1030 ms. Trace:
I20250112 02:52:48.245589 3352 tablet.cc:1721] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 02:52:48.246908 1173 tablet_bootstrap.cc:654] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:48.248131 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 9706187d3f9d4368a8e47962bc73ef3d from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:48.245218 1013 rpcz_store.cc:276] 0112 02:52:47.214803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.214862 (+ 59us) service_pool.cc:225] Handling call
0112 02:52:48.104605 (+889743us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x616000115c54. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:48.245126 (+140521us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":244682624}
I20250112 02:52:48.250267 446 master_service.cc:946] ReplaceTablet: received request to replace tablet b1cb48c936844d049a1c832b47986d73 from {username='slave'} at 127.0.0.1:40610
W20250112 02:52:48.250448 854 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.0.104.62:42341, user_credentials={real_user=slave}} blocked reactor thread for 59602.1us
I20250112 02:52:48.250654 3365 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:49624 (request call id 509) took 1598 ms. Trace:
I20250112 02:52:48.251528 938 tablet_copy_service.cc:342] P fe36951a83fb4e57b26e187edbcbfe3d: Request end of tablet copy session f1e775296eb746db8b318f17d26820e6-164f76edcf8f4bd498dcd2c25fd7477d received from {username='slave'} at 127.0.104.3:52739
I20250112 02:52:48.251645 938 tablet_copy_service.cc:434] P fe36951a83fb4e57b26e187edbcbfe3d: ending tablet copy session f1e775296eb746db8b318f17d26820e6-164f76edcf8f4bd498dcd2c25fd7477d on tablet 164f76edcf8f4bd498dcd2c25fd7477d with peer f1e775296eb746db8b318f17d26820e6
I20250112 02:52:48.251832 3111 raft_consensus.cc:2238] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:48.251987 3111 raft_consensus.cc:2267] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:48.253648 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_79dbb2fcd6fe4c939bc0637b4ddf1b1e" } modify_external_catalogs: true
I20250112 02:52:48.253998 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_79dbb2fcd6fe4c939bc0637b4ddf1b1e" } modify_external_catalogs: true
I20250112 02:52:48.255853 3352 tablet.cc:1721] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:52:48.258424 714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 48965.7us
I20250112 02:52:48.262223 3109 log.cc:1198] T 26577b614e32458e8bc5d6830c8c2e40 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/26577b614e32458e8bc5d6830c8c2e40
I20250112 02:52:48.262429 3110 raft_consensus.cc:2238] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:48.262635 3109 ts_tablet_manager.cc:1939] T 26577b614e32458e8bc5d6830c8c2e40 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:48.262778 3110 raft_consensus.cc:2267] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:48.263302 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 474) took 2825 ms. Trace:
I20250112 02:52:48.263478 3110 ts_tablet_manager.cc:1905] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:48.263401 3109 rpcz_store.cc:276] 0112 02:52:45.438012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.438066 (+ 54us) service_pool.cc:225] Handling call
0112 02:52:47.607277 (+2169211us) ts_tablet_manager.cc:1145] Deleting tablet 26577b614e32458e8bc5d6830c8c2e40
0112 02:52:47.607286 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.023163 (+415877us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.175340 (+152177us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.263265 (+ 87925us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2169005}
I20250112 02:52:48.263640 3111 ts_tablet_manager.cc:1905] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:48.263731 3109 tablet_replica.cc:331] T 61eac6b572ff47c185768bee7490edf7 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:48.266340 889 tablet_service.cc:1514] Processing DeleteTablet for tablet d1269a6ed80e46ecb7f4f87e5afb732f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:48.267982 3138 raft_consensus.cc:491] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:48.269265 714 leader_election.cc:304] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:48.270931 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 35169581764b416789c8981adc5b2e6b (DEFAULT_TABLE table=default.Table_479d0be56c26459c938d6fb9c52d4644 [id=71cb3aa27f724367a36b86d367dd15d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.271267 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35169581764b416789c8981adc5b2e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.272650 3370 tablet.cc:1721] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.278229 3106 raft_consensus.cc:2798] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:48.278390 3106 raft_consensus.cc:695] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:48.278602 3106 consensus_queue.cc:237] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.270161 3109 raft_consensus.cc:2238] T 61eac6b572ff47c185768bee7490edf7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:48.250746 3365 rpcz_store.cc:276] 0112 02:52:46.652277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.652327 (+ 50us) service_pool.cc:225] Handling call
0112 02:52:48.250639 (+1598312us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:46.653229 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 02:52:46.653319 (+ 90us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 02:52:46.653329 (+ 10us) alter_schema_op.cc:66] Acquired schema lock
0112 02:52:46.653338 (+ 9us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 02:52:46.653378 (+ 40us) alter_schema_op.cc:128] START. Timestamp: 1736650366653310
0112 02:52:46.653383 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:46.653564 (+ 181us) log.cc:844] Serialized 173 byte log entry
0112 02:52:47.972831 (+1319267us) op_driver.cc:464] REPLICATION: finished
0112 02:52:48.044826 (+ 71995us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 02:52:48.045294 (+ 468us) log.cc:844] Serialized 5 byte log entry
0112 02:52:48.225890 (+180596us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.250579 (+ 24689us) log.cc:844] Serialized 14 byte log entry
0112 02:52:48.250607 (+ 28us) alter_schema_op.cc:73] Released schema lock
0112 02:52:48.250611 (+ 4us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":71928,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":31017,"prepare.run_wall_time_us":700535,"replication_time_us":1319420,"thread_start_us":771704,"threads_started":2,"wal-append.queue_time_us":699784}]]}
I20250112 02:52:48.284219 3138 raft_consensus.cc:513] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.284518 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_16bf9ce62e194344833127d6803825f2" } new_table_name: "default.Table_f161f1352b5449ebbac77fcb007d79e0" modify_external_catalogs: true
I20250112 02:52:48.290025 3109 raft_consensus.cc:2267] T 61eac6b572ff47c185768bee7490edf7 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:48.290190 3365 tablet.cc:1721] T ca6f080e5fd84084b601b646d4e6c9e2 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:48.290622 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f16ec377f01484eae10e63490da143f" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:48.290915 910 raft_consensus.cc:2463] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:48.291303 713 leader_election.cc:304] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:48.293432 3138 leader_election.cc:290] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:48.293613 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 5cabc6fa3f93454aa503dbc5ee58659c (DEFAULT_TABLE table=default.Table_93c04d06452c40b680ed534a136f7eb8 [id=bba4dd86cfa6498598278742cbada067]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.294873 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cabc6fa3f93454aa503dbc5ee58659c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.296274 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a470f84d50df4cf0b4a1e5d248b4c32a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:48.296470 912 raft_consensus.cc:2463] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:48.298938 3352 tablet.cc:1721] T 4a57eb366f594f93a488c8250e195014 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.302520 3138 raft_consensus.cc:2798] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 02:52:48.302703 3138 raft_consensus.cc:491] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:48.302866 3138 raft_consensus.cc:3054] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Advancing to term 2
I20250112 02:52:48.303125 1177 tablet_bootstrap.cc:654] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:48.303963 714 consensus_peers.cc:487] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8 -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24ebf85f76f44633a20d97964638c0ac. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:48.310317 3109 ts_tablet_manager.cc:1905] T 61eac6b572ff47c185768bee7490edf7 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:48.313215 3111 ts_tablet_manager.cc:1918] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:48.313314 3111 log.cc:1198] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/e477fc157aed4563923d85d66014293a
I20250112 02:52:48.313644 3111 ts_tablet_manager.cc:1939] T e477fc157aed4563923d85d66014293a P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:48.313912 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 464) took 1701 ms. Trace:
I20250112 02:52:48.313983 3111 rpcz_store.cc:276] 0112 02:52:46.612511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.612552 (+ 41us) service_pool.cc:225] Handling call
0112 02:52:48.233767 (+1621215us) ts_tablet_manager.cc:1145] Deleting tablet e477fc157aed4563923d85d66014293a
0112 02:52:48.233774 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.296803 (+ 63029us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.313148 (+ 16345us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.313900 (+ 752us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26020,"tablet-delete.queue_time_us":1621072}
I20250112 02:52:48.314450 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet e477fc157aed4563923d85d66014293a (table default.Table_e495ee84abe3479c80c8ab2e14f86c96 [id=f5eb0a253e444d76978a22250c163a54]) successfully deleted
I20250112 02:52:48.316066 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 1692718d8c964829b0f9f31cfc0101b3 (DEFAULT_TABLE table=default.Table_6186d166c9594cfea3e160905f3737f7 [id=e8c858717b0a45ec99f7a800c94a7981]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.316367 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1692718d8c964829b0f9f31cfc0101b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.317799 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet acafc4f579534a69a8f307ca4cf72e12
I20250112 02:52:48.318322 454 catalog_manager.cc:6596] ReplaceTablet: tablet acafc4f579534a69a8f307ca4cf72e12 deleted and replaced by tablet a72ca53682fd4617aad9f8f9f3693998
I20250112 02:52:48.318598 454 master_service.cc:946] ReplaceTablet: received request to replace tablet a470f84d50df4cf0b4a1e5d248b4c32a from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:48.321517 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet acafc4f579534a69a8f307ca4cf72e12 with delete_type TABLET_DATA_DELETED (replaced by tablet a72ca53682fd4617aad9f8f9f3693998) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:48.322706 3355 tablet.cc:1721] T 46d1beac4410422bbf1e912d35999f3d P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:48.326645 885 tablet_service.cc:1514] Processing DeleteTablet for tablet acafc4f579534a69a8f307ca4cf72e12 with delete_type TABLET_DATA_DELETED (replaced by tablet a72ca53682fd4617aad9f8f9f3693998) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:48.327052 889 tablet_service.cc:1467] Processing CreateTablet for tablet 35169581764b416789c8981adc5b2e6b (DEFAULT_TABLE table=default.Table_479d0be56c26459c938d6fb9c52d4644 [id=71cb3aa27f724367a36b86d367dd15d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.327404 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35169581764b416789c8981adc5b2e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.327339 3365 tablet.cc:1721] T ead501d76bb1412e9839fb36b9911e6f P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:48.327800 888 tablet_service.cc:1467] Processing CreateTablet for tablet 28228d4200764cc68e83cc0f27b5e3a0 (DEFAULT_TABLE table=default.Table_1378877b3ffb4d5a95f50b55b1bf1297 [id=7041e2c37e5a4d09aef33319e18f210e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.328097 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28228d4200764cc68e83cc0f27b5e3a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.333385 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 28228d4200764cc68e83cc0f27b5e3a0 (DEFAULT_TABLE table=default.Table_1378877b3ffb4d5a95f50b55b1bf1297 [id=7041e2c37e5a4d09aef33319e18f210e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.333740 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28228d4200764cc68e83cc0f27b5e3a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.337584 3364 raft_consensus.cc:491] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:48.337697 3364 raft_consensus.cc:513] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.338229 3364 leader_election.cc:290] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:48.340498 3355 tablet.cc:1721] T 2052b4536f9745f0a785f330d706c1cf P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.342815 3370 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:49624 (request call id 527) took 1217 ms. Trace:
I20250112 02:52:48.342957 3370 rpcz_store.cc:276] 0112 02:52:47.125302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.128882 (+ 3580us) service_pool.cc:225] Handling call
0112 02:52:48.342778 (+1213896us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:47.353819 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 02:52:47.353935 (+ 116us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 02:52:47.353947 (+ 12us) alter_schema_op.cc:66] Acquired schema lock
0112 02:52:47.353956 (+ 9us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 02:52:47.353996 (+ 40us) alter_schema_op.cc:128] START. Timestamp: 1736650367353895
0112 02:52:47.354002 (+ 6us) op_driver.cc:348] REPLICATION: starting
0112 02:52:47.354198 (+ 196us) log.cc:844] Serialized 173 byte log entry
0112 02:52:48.081331 (+727133us) op_driver.cc:464] REPLICATION: finished
0112 02:52:48.272609 (+191278us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 02:52:48.273080 (+ 471us) log.cc:844] Serialized 5 byte log entry
0112 02:52:48.342460 (+ 69380us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.342703 (+ 243us) log.cc:844] Serialized 14 byte log entry
0112 02:52:48.342744 (+ 41us) alter_schema_op.cc:73] Released schema lock
0112 02:52:48.342748 (+ 4us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":189065,"prepare.queue_time_us":200935,"prepare.run_cpu_time_us":5712,"prepare.run_wall_time_us":591481,"replication_time_us":727300,"thread_start_us":756220,"threads_started":3,"wal-append.queue_time_us":588081}]]}
I20250112 02:52:48.345057 3370 tablet.cc:1721] T 2052b4536f9745f0a785f330d706c1cf P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.343690 3371 tablet.cc:1721] T 7320ac55dbfa47fe851fff94d32e1ffe P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.348203 893 tablet_service.cc:1467] Processing CreateTablet for tablet 1692718d8c964829b0f9f31cfc0101b3 (DEFAULT_TABLE table=default.Table_6186d166c9594cfea3e160905f3737f7 [id=e8c858717b0a45ec99f7a800c94a7981]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.348575 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1692718d8c964829b0f9f31cfc0101b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.348677 3335 tablet.cc:1716] T ca6f080e5fd84084b601b646d4e6c9e2 P fe36951a83fb4e57b26e187edbcbfe3d: Skipping requested alter to schema version 3, tablet already version 3
I20250112 02:52:48.350340 3365 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:49624 (request call id 555) took 1146 ms. Trace:
I20250112 02:52:48.350489 3365 rpcz_store.cc:276] 0112 02:52:47.213502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.315330 (+101828us) service_pool.cc:225] Handling call
0112 02:52:48.350304 (+1034974us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:47.355057 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 02:52:47.355108 (+ 51us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 02:52:47.355115 (+ 7us) alter_schema_op.cc:66] Acquired schema lock
0112 02:52:47.355121 (+ 6us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 02:52:47.355150 (+ 29us) alter_schema_op.cc:128] START. Timestamp: 1736650367355053
0112 02:52:47.355153 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:47.355319 (+ 166us) log.cc:844] Serialized 173 byte log entry
0112 02:52:48.178998 (+823679us) op_driver.cc:464] REPLICATION: finished
0112 02:52:48.327293 (+148295us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 02:52:48.337814 (+ 10521us) log.cc:844] Serialized 5 byte log entry
0112 02:52:48.344506 (+ 6692us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.344731 (+ 225us) log.cc:844] Serialized 14 byte log entry
0112 02:52:48.350281 (+ 5550us) alter_schema_op.cc:73] Released schema lock
0112 02:52:48.350285 (+ 4us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":148247,"prepare.queue_time_us":37679,"prepare.run_cpu_time_us":43085,"prepare.run_wall_time_us":661009,"replication_time_us":823807,"thread_start_us":660657,"threads_started":2,"wal-append.queue_time_us":615908}]]}
I20250112 02:52:48.353120 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_d9117b8cf65240dab1d9c1f61fe05192 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 2 got 1
I20250112 02:52:48.355747 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 470) took 1650 ms. Trace:
I20250112 02:52:48.355844 3111 rpcz_store.cc:276] 0112 02:52:46.705548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.705696 (+ 148us) service_pool.cc:225] Handling call
0112 02:52:48.355614 (+1649918us) ts_tablet_manager.cc:1145] Deleting tablet 17c54c5dc86446cf85974777c8145ef6
0112 02:52:48.355626 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.355729 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1649691}
I20250112 02:52:48.356319 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 471) took 1646 ms. Trace:
W20250112 02:52:48.356343 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 17c54c5dc86446cf85974777c8145ef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17c54c5dc86446cf85974777c8145ef6 already in progress: creating tablet
I20250112 02:52:48.356387 3111 rpcz_store.cc:276] 0112 02:52:46.709725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.710520 (+ 795us) service_pool.cc:225] Handling call
0112 02:52:48.356227 (+1645707us) ts_tablet_manager.cc:1145] Deleting tablet 8fe57d374eb44c3fb1532484e214b654
0112 02:52:48.356235 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.356306 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1645538}
W20250112 02:52:48.356642 854 leader_election.cc:343] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:48.356714 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 8fe57d374eb44c3fb1532484e214b654 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fe57d374eb44c3fb1532484e214b654 already in progress: creating tablet
I20250112 02:52:48.356736 854 leader_election.cc:304] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:48.356762 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 472) took 1645 ms. Trace:
I20250112 02:52:48.356813 3111 rpcz_store.cc:276] 0112 02:52:46.711159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.711213 (+ 54us) service_pool.cc:225] Handling call
0112 02:52:48.356679 (+1645466us) ts_tablet_manager.cc:1145] Deleting tablet 99407ead9cf547e3bbe78c0d0b327e6a
0112 02:52:48.356687 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.356748 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1645324}
W20250112 02:52:48.357101 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 99407ead9cf547e3bbe78c0d0b327e6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99407ead9cf547e3bbe78c0d0b327e6a already in progress: creating tablet
I20250112 02:52:48.357188 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 469) took 1652 ms. Trace:
I20250112 02:52:48.357246 3111 rpcz_store.cc:276] 0112 02:52:46.704559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.727152 (+ 22593us) service_pool.cc:225] Handling call
0112 02:52:48.357109 (+1629957us) ts_tablet_manager.cc:1145] Deleting tablet 8c12a1c5e3584450ac193743a0107e80
0112 02:52:48.357115 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.357176 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1629684}
I20250112 02:52:48.357487 3260 raft_consensus.cc:2743] T 5450777104da48f1821575658fd31afd P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:48.357565 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 8c12a1c5e3584450ac193743a0107e80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c12a1c5e3584450ac193743a0107e80 already in progress: creating tablet
I20250112 02:52:48.357601 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 477) took 1620 ms. Trace:
I20250112 02:52:48.354830 3109 ts_tablet_manager.cc:1918] T 61eac6b572ff47c185768bee7490edf7 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:48.357661 3111 rpcz_store.cc:276] 0112 02:52:46.737404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.737442 (+ 38us) service_pool.cc:225] Handling call
0112 02:52:48.357537 (+1620095us) ts_tablet_manager.cc:1145] Deleting tablet d63d411f906b4d4eb68343bdcc493e05
0112 02:52:48.357546 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.357590 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1619931}
I20250112 02:52:48.355531 3138 raft_consensus.cc:513] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:48.357950 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet d63d411f906b4d4eb68343bdcc493e05 because the tablet was not found. No further retry: Not found: Tablet not found: d63d411f906b4d4eb68343bdcc493e05
I20250112 02:52:48.358011 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 475) took 1620 ms. Trace:
W20250112 02:52:48.358006 854 leader_election.cc:343] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.358079 3111 rpcz_store.cc:276] 0112 02:52:46.737181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.737356 (+ 175us) service_pool.cc:225] Handling call
0112 02:52:48.357948 (+1620592us) ts_tablet_manager.cc:1145] Deleting tablet 62f4bd2df0ce45dabc55373f1154d3ac
0112 02:52:48.357956 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.357998 (+ 42us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1620153}
I20250112 02:52:48.358428 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 476) took 1621 ms. Trace:
W20250112 02:52:48.358410 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 62f4bd2df0ce45dabc55373f1154d3ac because the tablet was not found. No further retry: Not found: Tablet not found: 62f4bd2df0ce45dabc55373f1154d3ac
I20250112 02:52:48.358494 3111 rpcz_store.cc:276] 0112 02:52:46.737304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.737837 (+ 533us) service_pool.cc:225] Handling call
0112 02:52:48.358362 (+1620525us) ts_tablet_manager.cc:1145] Deleting tablet 54215c55966a46e1ab96e352c4e3764e
0112 02:52:48.358369 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.358416 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1620357}
W20250112 02:52:48.358785 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 54215c55966a46e1ab96e352c4e3764e because the tablet was not found. No further retry: Not found: Tablet not found: 54215c55966a46e1ab96e352c4e3764e
W20250112 02:52:48.364434 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 1a7717cdcb8a4f47b11ba7af54b57e3c because the tablet was not found. No further retry: Not found: Tablet not found: 1a7717cdcb8a4f47b11ba7af54b57e3c
I20250112 02:52:48.363641 3109 log.cc:1198] T 61eac6b572ff47c185768bee7490edf7 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/61eac6b572ff47c185768bee7490edf7
W20250112 02:52:48.365787 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 4cfe1a1595014c27bbd350cec169d71a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cfe1a1595014c27bbd350cec169d71a already in progress: creating tablet
I20250112 02:52:48.365865 3138 leader_election.cc:290] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 2 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:48.366235 3109 ts_tablet_manager.cc:1939] T 61eac6b572ff47c185768bee7490edf7 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
W20250112 02:52:48.366226 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f52b838214c54ac69ad30dac6e56669a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f52b838214c54ac69ad30dac6e56669a already in progress: creating tablet
I20250112 02:52:48.365712 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f16ec377f01484eae10e63490da143f" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:48.367161 909 raft_consensus.cc:3054] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Advancing to term 2
W20250112 02:52:48.368091 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6b0bcd74d7f041d8b84bc5fe08b18093 has no consensus state
I20250112 02:52:48.369092 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_f180982a92d4460d94f3f67eccd27009" } modify_external_catalogs: true
I20250112 02:52:48.369357 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_f180982a92d4460d94f3f67eccd27009" } modify_external_catalogs: true
I20250112 02:52:48.369688 886 tablet_service.cc:1467] Processing CreateTablet for tablet f5492416a0544951bcbf8a4f84942e66 (DEFAULT_TABLE table=default.Table_f8ab3098ff7f4c479588888eac14e9a7 [id=a1bcfb4b28f54654ac905844df2ef79d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.370132 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5492416a0544951bcbf8a4f84942e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.372538 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_219c6fd235a2425dbbe5645e1660f7f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:48.373179 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_219c6fd235a2425dbbe5645e1660f7f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.374491 3365 tablet.cc:1721] T 42444e99aba84a7b873bc8cdf954a6dc P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.375015 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 474) took 1638 ms. Trace:
I20250112 02:52:48.375161 3111 rpcz_store.cc:276] 0112 02:52:46.737044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.739069 (+ 2025us) service_pool.cc:225] Handling call
0112 02:52:48.374907 (+1635838us) ts_tablet_manager.cc:1145] Deleting tablet 13578542e9e741e8817e0ee63b10f1a1
0112 02:52:48.374920 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.374992 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1635608}
I20250112 02:52:48.375710 3111 tablet_replica.cc:331] T 09900b42e8a443e680cc35c96a1dbbb7 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
W20250112 02:52:48.375818 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 13578542e9e741e8817e0ee63b10f1a1 because the tablet was not found. No further retry: Not found: Tablet not found: 13578542e9e741e8817e0ee63b10f1a1
I20250112 02:52:48.376049 3111 raft_consensus.cc:2238] T 09900b42e8a443e680cc35c96a1dbbb7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:48.376255 3111 raft_consensus.cc:2267] T 09900b42e8a443e680cc35c96a1dbbb7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:48.358985 3374 tablet.cc:1721] T 4a57eb366f594f93a488c8250e195014 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.376791 3370 tablet.cc:1721] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 02:52:48.376781 3371 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:49624 (request call id 528) took 1240 ms. Trace:
I20250112 02:52:48.378248 888 tablet_service.cc:1467] Processing CreateTablet for tablet 5cabc6fa3f93454aa503dbc5ee58659c (DEFAULT_TABLE table=default.Table_93c04d06452c40b680ed534a136f7eb8 [id=bba4dd86cfa6498598278742cbada067]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.378571 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cabc6fa3f93454aa503dbc5ee58659c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:48.378650 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d95f98ae7bdf42518bbff55e257fea88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d95f98ae7bdf42518bbff55e257fea88 already in progress: creating tablet
W20250112 02:52:48.378942 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet be856e4b913e40d58a3f3ba54698cc5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be856e4b913e40d58a3f3ba54698cc5f already in progress: creating tablet
W20250112 02:52:48.379218 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 3d662e0ce13f447595e5f92b94de3018 because the tablet was not found. No further retry: Not found: Tablet not found: 3d662e0ce13f447595e5f92b94de3018
I20250112 02:52:48.379900 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 473) took 2929 ms. Trace:
I20250112 02:52:48.358263 854 leader_election.cc:304] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:48.383275 3260 raft_consensus.cc:2743] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:48.383829 854 leader_election.cc:343] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.384027 854 leader_election.cc:304] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:48.384846 3260 raft_consensus.cc:2743] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:48.393517 1181 tablet_bootstrap.cc:492] T 3dab0466476c4a9aa265271c1abd91a4 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:48.394019 1181 ts_tablet_manager.cc:1397] T 3dab0466476c4a9aa265271c1abd91a4 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.533s user 0.013s sys 0.014s
W20250112 02:52:48.399670 854 leader_election.cc:343] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.399732 3111 ts_tablet_manager.cc:1905] T 09900b42e8a443e680cc35c96a1dbbb7 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:48.399778 854 leader_election.cc:304] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:48.400013 3260 raft_consensus.cc:2743] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:48.400566 854 leader_election.cc:343] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.400653 1181 raft_consensus.cc:357] T 3dab0466476c4a9aa265271c1abd91a4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.401525 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 26577b614e32458e8bc5d6830c8c2e40 (table default.Table_03e3448d809c4fc792d0bff94a713044 [id=5f3cc73b47414c438591107e35a12472]) successfully deleted
I20250112 02:52:48.401775 3364 raft_consensus.cc:491] T ea33053c62dc4707b759d5c1756ab4bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:48.401639 1181 raft_consensus.cc:383] T 3dab0466476c4a9aa265271c1abd91a4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:48.401891 3364 raft_consensus.cc:513] T ea33053c62dc4707b759d5c1756ab4bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.402443 3364 leader_election.cc:290] T ea33053c62dc4707b759d5c1756ab4bd P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:48.401970 1181 raft_consensus.cc:738] T 3dab0466476c4a9aa265271c1abd91a4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:48.403133 1177 tablet_bootstrap.cc:492] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:48.403208 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f16ec377f01484eae10e63490da143f" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:48.375788 909 raft_consensus.cc:2463] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 2.
I20250112 02:52:48.403291 3109 rpcz_store.cc:276] 0112 02:52:45.437897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.441459 (+ 3562us) service_pool.cc:225] Handling call
0112 02:52:48.263671 (+2822212us) ts_tablet_manager.cc:1145] Deleting tablet 61eac6b572ff47c185768bee7490edf7
0112 02:52:48.263680 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.351701 (+ 88021us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.354743 (+ 3042us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.367051 (+ 12308us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2821983}
I20250112 02:52:48.405150 713 leader_election.cc:304] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:48.405335 1177 ts_tablet_manager.cc:1397] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.230s user 0.001s sys 0.009s
I20250112 02:52:48.405743 3296 raft_consensus.cc:2798] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 02:52:48.405891 3296 raft_consensus.cc:695] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [term 2 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:48.406224 1177 raft_consensus.cc:357] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.406414 1177 raft_consensus.cc:383] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:48.406503 1177 raft_consensus.cc:738] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:48.406683 1177 consensus_queue.cc:260] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.407207 1177 ts_tablet_manager.cc:1428] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:48.407450 1177 tablet_bootstrap.cc:492] T d95f98ae7bdf42518bbff55e257fea88 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:48.407971 3109 tablet_replica.cc:331] T e4d42f1611de49d6b6e5aaa9be907054 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:48.406230 3296 consensus_queue.cc:237] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.411258 1181 consensus_queue.cc:260] T 3dab0466476c4a9aa265271c1abd91a4 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.411669 3352 tablet.cc:1721] T 5a3af0ea4e954053ae89814f3e74e6b7 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.412505 1181 ts_tablet_manager.cc:1428] T 3dab0466476c4a9aa265271c1abd91a4 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20250112 02:52:48.412907 1181 tablet_bootstrap.cc:492] T fc2bc28710d047cf92f6e92509c2bbaf P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:48.415128 3110 ts_tablet_manager.cc:1918] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 02:52:48.415318 3110 log.cc:1198] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/09d06e2e8c13464db8cf801a292bdc99
I20250112 02:52:48.416887 3109 raft_consensus.cc:2238] T e4d42f1611de49d6b6e5aaa9be907054 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:48.421371 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bf8a907f0af4e24819d1f00e40885ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:48.421813 453 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 91ce82a8f7e644858a8bdfb7d52bdbbf
I20250112 02:52:48.421872 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet a470f84d50df4cf0b4a1e5d248b4c32a
I20250112 02:52:48.422175 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_8bb2bb131d6b4262b964926902af01d3" } new_table_name: "default.Table_7a6fda16f2604cf4b5676bd749f16989" modify_external_catalogs: true
I20250112 02:52:48.422346 454 catalog_manager.cc:6596] ReplaceTablet: tablet a470f84d50df4cf0b4a1e5d248b4c32a deleted and replaced by tablet ff4c955aa10846f7bcbe0c4097fa72fa
I20250112 02:52:48.422657 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_33132fb47fe34d5d81a90474dec05f14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:48.423116 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33132fb47fe34d5d81a90474dec05f14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.423553 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet a470f84d50df4cf0b4a1e5d248b4c32a with delete_type TABLET_DATA_DELETED (replaced by tablet ff4c955aa10846f7bcbe0c4097fa72fa) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:48.423859 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet ecfc7d8ba5e4406ebc61f37e6d500473
I20250112 02:52:48.424501 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_65dfa36d9f664916bf107d10764f88cd" } modify_external_catalogs: true
I20250112 02:52:48.424682 1177 tablet_bootstrap.cc:654] T d95f98ae7bdf42518bbff55e257fea88 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:48.424680 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_65dfa36d9f664916bf107d10764f88cd" } modify_external_catalogs: true
I20250112 02:52:48.426189 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet ecfc7d8ba5e4406ebc61f37e6d500473 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:48.426445 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_dc630a24c2df45b092d2118e2fa378dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:48.427037 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc630a24c2df45b092d2118e2fa378dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.428030 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 9706187d3f9d4368a8e47962bc73ef3d
I20250112 02:52:48.428112 3260 raft_consensus.cc:491] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:48.428231 3260 raft_consensus.cc:513] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.428531 449 catalog_manager.cc:6596] ReplaceTablet: tablet 9706187d3f9d4368a8e47962bc73ef3d deleted and replaced by tablet 60042e463741482190544e86cc8512c3
I20250112 02:52:48.428687 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 9706187d3f9d4368a8e47962bc73ef3d with delete_type TABLET_DATA_DELETED (replaced by tablet 60042e463741482190544e86cc8512c3) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:48.379540 3371 rpcz_store.cc:276] 0112 02:52:47.125416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.127380 (+ 1964us) service_pool.cc:225] Handling call
0112 02:52:48.365556 (+1238176us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:47.299254 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 02:52:47.299311 (+ 57us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 02:52:47.299319 (+ 8us) alter_schema_op.cc:66] Acquired schema lock
0112 02:52:47.299322 (+ 3us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 02:52:47.299347 (+ 25us) alter_schema_op.cc:128] START. Timestamp: 1736650367299265
0112 02:52:47.299349 (+ 2us) op_driver.cc:348] REPLICATION: starting
0112 02:52:47.325518 (+ 26169us) log.cc:844] Serialized 173 byte log entry
0112 02:52:48.194375 (+868857us) op_driver.cc:464] REPLICATION: finished
0112 02:52:48.343646 (+149271us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 02:52:48.345698 (+ 2052us) log.cc:844] Serialized 5 byte log entry
0112 02:52:48.363425 (+ 17727us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.365355 (+ 1930us) log.cc:844] Serialized 14 byte log entry
0112 02:52:48.365524 (+ 169us) alter_schema_op.cc:73] Released schema lock
0112 02:52:48.365529 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":149209,"prepare.queue_time_us":171608,"prepare.run_cpu_time_us":29554,"prepare.run_wall_time_us":645079,"replication_time_us":895000,"thread_start_us":941171,"threads_started":2,"wal-append.queue_time_us":610386}]]}
I20250112 02:52:48.428772 3260 leader_election.cc:290] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:48.428845 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_fa94e8bd95fe476ea93f58a62847d3ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:48.428990 3371 tablet.cc:1721] T 7cd9985cd20148f195a857c478b0755f P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 02:52:48.429252 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa94e8bd95fe476ea93f58a62847d3ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.430408 3109 raft_consensus.cc:2267] T e4d42f1611de49d6b6e5aaa9be907054 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:48.430810 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_c2314b98707643e39774d4255163325d" } new_table_name: "default.Table_f2cc2e09085e4af0a88b618385e5f50f" modify_external_catalogs: true
I20250112 02:52:48.437477 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 9706187d3f9d4368a8e47962bc73ef3d with delete_type TABLET_DATA_DELETED (replaced by tablet 60042e463741482190544e86cc8512c3) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:48.437968 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ddc025a8314f6f84d229191eb56692" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:48.438398 893 tablet_service.cc:1514] Processing DeleteTablet for tablet a470f84d50df4cf0b4a1e5d248b4c32a with delete_type TABLET_DATA_DELETED (replaced by tablet ff4c955aa10846f7bcbe0c4097fa72fa) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:48.438459 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet b1cb48c936844d049a1c832b47986d73
I20250112 02:52:48.438858 446 catalog_manager.cc:6596] ReplaceTablet: tablet b1cb48c936844d049a1c832b47986d73 deleted and replaced by tablet ca9a18d7e84e44369c29593021870197
W20250112 02:52:48.439450 855 leader_election.cc:343] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.440011 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet b1cb48c936844d049a1c832b47986d73 with delete_type TABLET_DATA_DELETED (replaced by tablet ca9a18d7e84e44369c29593021870197) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:48.442620 893 tablet_service.cc:1514] Processing DeleteTablet for tablet b1cb48c936844d049a1c832b47986d73 with delete_type TABLET_DATA_DELETED (replaced by tablet ca9a18d7e84e44369c29593021870197) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:48.444759 889 tablet_service.cc:1514] Processing DeleteTablet for tablet ecfc7d8ba5e4406ebc61f37e6d500473 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:48.445470 3110 ts_tablet_manager.cc:1939] T 09d06e2e8c13464db8cf801a292bdc99 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:48.445557 3109 ts_tablet_manager.cc:1905] T e4d42f1611de49d6b6e5aaa9be907054 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:48.445897 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_fce3401db3744b4a9f293c9ca3923013"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:48.446336 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fce3401db3744b4a9f293c9ca3923013 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.446643 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 364) took 2769 ms. Trace:
I20250112 02:52:48.446781 3110 rpcz_store.cc:276] 0112 02:52:45.677385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.677741 (+ 356us) service_pool.cc:225] Handling call
0112 02:52:48.149342 (+2471601us) ts_tablet_manager.cc:1145] Deleting tablet 09d06e2e8c13464db8cf801a292bdc99
0112 02:52:48.149351 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.306054 (+156703us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.415033 (+108979us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.446624 (+ 31591us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":112834,"tablet-delete.queue_time_us":2471421}
I20250112 02:52:48.447309 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 09d06e2e8c13464db8cf801a292bdc99 (table default.Table_2c52fead633649dd869763e5aa532be7 [id=e3bb489c53454222a79702b798c0c2f1]) successfully deleted
I20250112 02:52:48.447451 3110 tablet_replica.cc:331] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:48.447916 3110 raft_consensus.cc:2238] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:48.448324 3110 raft_consensus.cc:2267] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:48.448410 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_4ce8a06de05a4423a9744ed9fb6c2ec4 [id=70e9b6c9e20f40eab186f042f21ad2ec] alter complete (version 1)
I20250112 02:52:48.448482 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 0925ba2197bb461190b832d062567040
I20250112 02:52:48.449472 3110 ts_tablet_manager.cc:1905] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:48.449618 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 0925ba2197bb461190b832d062567040 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:48.450008 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 37862023095644bab2403576e8cb8c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:48.450131 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_c7198ffff87c46c8a11305b0fc10a2ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:48.450287 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 0925ba2197bb461190b832d062567040 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:48.450472 3260 raft_consensus.cc:491] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 02:52:48.450546 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7198ffff87c46c8a11305b0fc10a2ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.450575 3260 raft_consensus.cc:513] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.451179 3260 leader_election.cc:290] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:48.451586 713 consensus_peers.cc:487] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:48.451766 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d7774d4a284d24b833d50809996fc2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:48.452384 855 leader_election.cc:343] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.454602 3374 tablet.cc:1721] T 79eb61cedd544c49869471eebe2fd237 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:48.456635 1181 tablet_bootstrap.cc:654] T fc2bc28710d047cf92f6e92509c2bbaf P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:48.400776 854 leader_election.cc:304] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:48.457800 3260 raft_consensus.cc:2743] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:48.458112 854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 57548.8us
I20250112 02:52:48.458345 854 leader_election.cc:304] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:48.460021 3260 raft_consensus.cc:2798] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:48.460107 3260 raft_consensus.cc:491] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:48.460172 3260 raft_consensus.cc:3054] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:48.463066 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_4965c2235c4e422889169067ff632d29" } new_table_name: "default.Table_04503e959a3c418a92dbb16260fbdc5c" modify_external_catalogs: true
I20250112 02:52:48.463351 3260 raft_consensus.cc:513] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.463981 3260 leader_election.cc:290] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:48.472843 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5b78e2f2303348dea1ca11c1cfd24e3f consensus state has no leader
I20250112 02:52:48.473196 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_dec5d4628a934e56a6203c75fc448dc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:48.473372 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet ee0c474eedcd459697261747e802279e
W20250112 02:52:48.473655 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dec5d4628a934e56a6203c75fc448dc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.474056 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_c11c44af47604f9589b15d5c6a97c255" } modify_external_catalogs: true
I20250112 02:52:48.474243 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_c11c44af47604f9589b15d5c6a97c255" } modify_external_catalogs: true
I20250112 02:52:48.475009 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet ee0c474eedcd459697261747e802279e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:48.475885 888 tablet_service.cc:1514] Processing DeleteTablet for tablet ee0c474eedcd459697261747e802279e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:38786
W20250112 02:52:48.480432 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3869ad978191401a801b177eda846aba has no consensus state
W20250112 02:52:48.485078 854 leader_election.cc:343] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.486012 854 leader_election.cc:304] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:48.486866 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ded8bcdda914ea6bc4f9bb92d413dee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:48.487282 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a470f84d50df4cf0b4a1e5d248b4c32a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:48.488034 3000 raft_consensus.cc:2743] T dcb7e0181cf74bbea6036c106516a7b3 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:48.488286 855 leader_election.cc:343] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.492237 989 leader_election.cc:304] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:48.492542 3364 raft_consensus.cc:2798] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:48.492660 3364 raft_consensus.cc:695] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:48.492813 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_77741e089cd44e608b680bec456ef0ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:48.493314 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77741e089cd44e608b680bec456ef0ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.493229 3364 consensus_queue.cc:237] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.495757 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_36ee8cc4670048c6b9c96ac874785d48 [id=e1779f88b20942eaa2c640ac24946fff] alter complete (version 1)
I20250112 02:52:48.496417 3245 tablet_copy_client.cc:538] T ca6f080e5fd84084b601b646d4e6c9e2 P f1e775296eb746db8b318f17d26820e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 02:52:48.499404 3245 tablet_bootstrap.cc:492] T ca6f080e5fd84084b601b646d4e6c9e2 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:48.500006 3282 ts_tablet_manager.cc:927] T 896908b2726c40f9aabdfc9a6b3283f7 P f1e775296eb746db8b318f17d26820e6: Initiating tablet copy from peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:48.503293 989 leader_election.cc:304] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:48.503543 3158 raft_consensus.cc:2798] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:48.503619 3158 raft_consensus.cc:491] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:48.503696 3158 raft_consensus.cc:3054] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:48.504699 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 41d5799b5b36487da42dc43b1c70ae26 (table default.Table_ecbd292479064383a2b82283bbcfc614 [id=e292141365e9453fa1816e0f313b2300]). Expected version 1 got 0
I20250112 02:52:48.504984 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_d9117b8cf65240dab1d9c1f61fe05192 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 2 got 0
I20250112 02:52:48.505226 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_67b6f55a9ba84491bdda41cdf82c25fc [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 4 got 0
I20250112 02:52:48.505389 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 79eb61cedd544c49869471eebe2fd237 (table default.Table_f957cb19a384456eb663c27c470cf702 [id=3a629cb4a4104e30b0beb8f729ba0a22]). Expected version 2 got 1
I20250112 02:52:48.505527 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet e933b58dab0b45e8bf1c49eebb1c53e3 (table default.Table_f161f1352b5449ebbac77fcb007d79e0 [id=e184c10af82240f5b1e29fc226269bf0]). Expected version 2 got 1
I20250112 02:52:48.505843 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_a9907ed8bfe14625977ec98b67e9d599 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 2 got 0
I20250112 02:52:48.506073 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:48.506255 449 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet ca2165362ee541adac724b63f39b1489 (table default.Table_8ffa45da6c19440cbac5f82e79c6641b [id=6402abce8d1d45c182cc989490c6a5f0]). Expected version 3 got 0
I20250112 02:52:48.506556 3158 raft_consensus.cc:513] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.506747 3111 ts_tablet_manager.cc:1918] T 09900b42e8a443e680cc35c96a1dbbb7 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:48.506839 3111 log.cc:1198] T 09900b42e8a443e680cc35c96a1dbbb7 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/09900b42e8a443e680cc35c96a1dbbb7
I20250112 02:52:48.507192 3158 leader_election.cc:290] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:48.508263 3266 consensus_queue.cc:1035] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:48.511713 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bd1fb8d69bd411bb02fdafa6ba720f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:48.512148 3282 tablet_copy_client.cc:323] T 896908b2726c40f9aabdfc9a6b3283f7 P f1e775296eb746db8b318f17d26820e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.104.1:35007
I20250112 02:52:48.513432 3353 raft_consensus.cc:491] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:48.513602 3353 raft_consensus.cc:513] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.513897 3273 raft_consensus.cc:491] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:48.514056 3273 raft_consensus.cc:513] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.514189 3353 leader_election.cc:290] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:48.515777 989 leader_election.cc:343] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 83c04f1a2cb04950a9f6170e252c19b4
I20250112 02:52:48.516765 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d7774d4a284d24b833d50809996fc2" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:48.517129 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a470f84d50df4cf0b4a1e5d248b4c32a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:48.517289 1046 raft_consensus.cc:2388] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fe36951a83fb4e57b26e187edbcbfe3d in current term 1: Already voted for candidate f1e775296eb746db8b318f17d26820e6 in this term.
I20250112 02:52:48.517604 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ddc025a8314f6f84d229191eb56692" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:48.507192 3111 ts_tablet_manager.cc:1939] T 09900b42e8a443e680cc35c96a1dbbb7 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:48.525671 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 485) took 1660 ms. Trace:
I20250112 02:52:48.526379 3111 rpcz_store.cc:276] 0112 02:52:46.865493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.865540 (+ 47us) service_pool.cc:225] Handling call
0112 02:52:48.375662 (+1510122us) ts_tablet_manager.cc:1145] Deleting tablet 09900b42e8a443e680cc35c96a1dbbb7
0112 02:52:48.375671 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.502440 (+126769us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.506664 (+ 4224us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.525653 (+ 18989us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1509913}
I20250112 02:52:48.527179 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 487) took 1645 ms. Trace:
I20250112 02:52:48.527343 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 09900b42e8a443e680cc35c96a1dbbb7 (table default.Table_5e1a058646834ea99def3d8262bee81b [id=ec894df00ecb4b62881cbc11cdede0a7]) successfully deleted
I20250112 02:52:48.527890 3111 rpcz_store.cc:276] 0112 02:52:46.881625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.881738 (+ 113us) service_pool.cc:225] Handling call
0112 02:52:48.527099 (+1645361us) ts_tablet_manager.cc:1145] Deleting tablet 61eac6b572ff47c185768bee7490edf7
0112 02:52:48.527108 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.527164 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1645102}
I20250112 02:52:48.528487 3111 tablet_replica.cc:331] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:48.529193 3111 raft_consensus.cc:2238] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:48.531121 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 2d3d291ca3494f05ac23c7fe293e0bd9
I20250112 02:52:48.531929 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_c0a21e51b1284e2ca2d529c6f24c81fa [id=f836cdee435949d387336c9a52b6ed1b] alter complete (version 1)
I20250112 02:52:48.529847 3111 raft_consensus.cc:2267] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:48.532464 3111 ts_tablet_manager.cc:1905] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:48.536747 441 catalog_manager.cc:4839] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 61eac6b572ff47c185768bee7490edf7 because the tablet was not found. No further retry: Not found: Tablet not found: 61eac6b572ff47c185768bee7490edf7
I20250112 02:52:48.538424 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3d291ca3494f05ac23c7fe293e0bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:48.538796 895 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38786 (request call id 450) took 1076 ms. Trace:
I20250112 02:52:48.545158 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c04f1a2cb04950a9f6170e252c19b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:34 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:48.545657 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 93aba9b4d1dd485cb62c4827c916a1d5 with delete_type TABLET_DATA_DELETED (replaced by tablet e3454657cbd74544b0ae962736e4b7b5) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:48.546159 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3d291ca3494f05ac23c7fe293e0bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:48.548759 3273 leader_election.cc:290] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:48.538949 895 rpcz_store.cc:276] 0112 02:52:47.462209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.462252 (+ 43us) service_pool.cc:225] Handling call
0112 02:52:47.462603 (+ 351us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 02:52:47.462622 (+ 19us) ts_tablet_manager.cc:676] Creating new metadata...
0112 02:52:48.467488 (+1004866us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.496693 (+ 29205us) tablet_replica.cc:206] Creating consensus instance
0112 02:52:48.538779 (+ 42086us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:52:48.568092 912 raft_consensus.cc:1235] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d [term 2 FOLLOWER]: Rejecting Update request from peer f1e775296eb746db8b318f17d26820e6 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250112 02:52:48.568348 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a470f84d50df4cf0b4a1e5d248b4c32a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:48.568517 747 tablet_service.cc:1514] Processing DeleteTablet for tablet d1269a6ed80e46ecb7f4f87e5afb732f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:48.568580 919 raft_consensus.cc:2388] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f1e775296eb746db8b318f17d26820e6 in current term 1: Already voted for candidate fe36951a83fb4e57b26e187edbcbfe3d in this term.
I20250112 02:52:48.569569 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 38e4a69f470a4d62b865e2dd594fab63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:48.570058 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ecf730b6985404a8134b497a4b792c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:48.570175 743 tablet_service.cc:1467] Processing CreateTablet for tablet e438e49f1e5345a78e11b66ac8e95d28 (DEFAULT_TABLE table=default.Table_80b349ce76644bcabf626e044231436a [id=ba3ca203a209474184c7763dabea3f5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.570453 753 tablet_service.cc:1467] Processing CreateTablet for tablet f5492416a0544951bcbf8a4f84942e66 (DEFAULT_TABLE table=default.Table_f8ab3098ff7f4c479588888eac14e9a7 [id=a1bcfb4b28f54654ac905844df2ef79d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.570876 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 164f76edcf8f4bd498dcd2c25fd7477d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:48.571705 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e438e49f1e5345a78e11b66ac8e95d28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.571707 760 tablet_service.cc:1467] Processing CreateTablet for tablet 35169581764b416789c8981adc5b2e6b (DEFAULT_TABLE table=default.Table_479d0be56c26459c938d6fb9c52d4644 [id=71cb3aa27f724367a36b86d367dd15d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.571985 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5492416a0544951bcbf8a4f84942e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.572147 758 tablet_service.cc:1514] Processing DeleteTablet for tablet acafc4f579534a69a8f307ca4cf72e12 with delete_type TABLET_DATA_DELETED (replaced by tablet a72ca53682fd4617aad9f8f9f3693998) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:48.572815 742 tablet_service.cc:1514] Processing DeleteTablet for tablet ecfc7d8ba5e4406ebc61f37e6d500473 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:48.573040 759 tablet_service.cc:1514] Processing DeleteTablet for tablet a18c4709d4a845309968abafac18f58f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:48.573487 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ea00cab2f6f44b58d82c02a07513416 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:48.574273 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1123077650c4e89bffddd6d1264e428" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:48.575565 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 9706187d3f9d4368a8e47962bc73ef3d with delete_type TABLET_DATA_DELETED (replaced by tablet 60042e463741482190544e86cc8512c3) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:48.575605 745 tablet_service.cc:1467] Processing CreateTablet for tablet 4af0d07a878c42f5a15ff9b8a2295f30 (DEFAULT_TABLE table=default.Table_b910dc09eda7442ea8f6df224e60e7db [id=1b55644d4bb14110a8c37d9a0dd2eccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.575951 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4af0d07a878c42f5a15ff9b8a2295f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.576604 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 68e480204ee94a44a1fef701ef5541f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:47 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:48.576885 741 tablet_service.cc:1467] Processing CreateTablet for tablet 5cabc6fa3f93454aa503dbc5ee58659c (DEFAULT_TABLE table=default.Table_93c04d06452c40b680ed534a136f7eb8 [id=bba4dd86cfa6498598278742cbada067]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.578373 755 tablet_service.cc:1467] Processing CreateTablet for tablet 81bad885cbbf4a8b9a06c8a066f25e4c (DEFAULT_TABLE table=default.Table_a6857eb18f404b15a97dade22ad383c1 [id=0f032dab356b4f1fb540c52e22153f99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.583498 1173 tablet_bootstrap.cc:492] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:48.583636 1173 ts_tablet_manager.cc:1397] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.441s user 0.015s sys 0.015s
I20250112 02:52:48.584358 1173 raft_consensus.cc:357] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.584530 1173 raft_consensus.cc:383] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:48.584578 1173 raft_consensus.cc:738] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:48.584969 3348 tablet.cc:1721] T 7320ac55dbfa47fe851fff94d32e1ffe P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.586934 748 tablet_service.cc:1467] Processing CreateTablet for tablet ad7b2e22c9584f55b5a4c996bd305e13 (DEFAULT_TABLE table=default.Table_518ec5f7946342419e87f093585388fe [id=55b32a6c95f94c2a8bf3f216238cdc92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.587873 757 tablet_service.cc:1467] Processing CreateTablet for tablet 1692718d8c964829b0f9f31cfc0101b3 (DEFAULT_TABLE table=default.Table_6186d166c9594cfea3e160905f3737f7 [id=e8c858717b0a45ec99f7a800c94a7981]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.589972 756 tablet_service.cc:1514] Processing DeleteTablet for tablet ee0c474eedcd459697261747e802279e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:48.590034 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35169581764b416789c8981adc5b2e6b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:48.590313 855 leader_election.cc:343] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:48.591609 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:35007, user_credentials={real_user=slave}} blocked reactor thread for 385297us
I20250112 02:52:48.591665 758 tablet_service.cc:1514] Processing DeleteTablet for tablet a470f84d50df4cf0b4a1e5d248b4c32a with delete_type TABLET_DATA_DELETED (replaced by tablet ff4c955aa10846f7bcbe0c4097fa72fa) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:48.591773 752 tablet_service.cc:1467] Processing CreateTablet for tablet 28228d4200764cc68e83cc0f27b5e3a0 (DEFAULT_TABLE table=default.Table_1378877b3ffb4d5a95f50b55b1bf1297 [id=7041e2c37e5a4d09aef33319e18f210e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:48.592608 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f16ec377f01484eae10e63490da143f" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:48.591589 744 tablet_service.cc:1514] Processing DeleteTablet for tablet b1cb48c936844d049a1c832b47986d73 with delete_type TABLET_DATA_DELETED (replaced by tablet ca9a18d7e84e44369c29593021870197) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:48.594110 991 consensus_peers.cc:487] T 54d3664071b146a3ac7c6885f9bda789 P f1e775296eb746db8b318f17d26820e6 -> Peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Couldn't send request to peer 289afc116aa74408aa5bff74e11c50d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:48.600122 3348 tablet.cc:1721] T e933b58dab0b45e8bf1c49eebb1c53e3 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.601684 3273 raft_consensus.cc:491] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:48.601812 3273 raft_consensus.cc:513] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.602383 3273 leader_election.cc:290] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:48.545945 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_bb0a4f973f234c80954ac449b0c6e88a [id=f95ac050a4b840a19a8175caade87d48] alter complete (version 3)
I20250112 02:52:48.610716 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_add1776e850b4a1bbd63f05707b51873" } new_table_name: "default.Table_e373fd8fe38d400f9bc33937fe89628b" modify_external_catalogs: true
I20250112 02:52:48.613565 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_baf3e4e049884072afa4b4cafe8ce3fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:48.614140 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_baf3e4e049884072afa4b4cafe8ce3fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.616896 3245 tablet_bootstrap.cc:492] T ca6f080e5fd84084b601b646d4e6c9e2 P f1e775296eb746db8b318f17d26820e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250112 02:52:48.617354 3245 tablet_bootstrap.cc:492] T ca6f080e5fd84084b601b646d4e6c9e2 P f1e775296eb746db8b318f17d26820e6: Bootstrap complete.
I20250112 02:52:48.617561 3245 ts_tablet_manager.cc:1397] T ca6f080e5fd84084b601b646d4e6c9e2 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.118s user 0.005s sys 0.006s
I20250112 02:52:48.617820 3273 raft_consensus.cc:491] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:48.617944 3273 raft_consensus.cc:513] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.618595 3273 leader_election.cc:290] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:48.620976 1014 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:50840 (request call id 136) took 1103 ms. Trace:
I20250112 02:52:48.621062 1014 rpcz_store.cc:276] 0112 02:52:47.517516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.519206 (+ 1690us) service_pool.cc:225] Handling call
0112 02:52:48.315059 (+795853us) spinlock_profiling.cc:243] Waited 92.8 ms on lock 0x616000280954. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:48.606243 (+291184us) spinlock_profiling.cc:243] Waited 27.5 ms on lock 0x616000133354. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:48.618174 (+ 11931us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x616000117454. stack: 00007f2e7ff2dfd6 00007f2e7ff2dd90 00007f2e7ef0d48d 000000000096d544 000000000096d50d 000000000096ceda 00007f2e8b7a0fd0 00007f2e8f4ddb3f 00007f2e89e388e6 00007f2e89e38615 00007f2e86c9530e 00007f2e86c941ee 00007f2e86c9a425 00007f2e86c9ad4f 00007f2e86c9ab78 0000000000a71ecc
0112 02:52:48.620956 (+ 2782us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":302338560}
I20250112 02:52:48.626132 3310 tablet_copy_client.cc:538] T 5804fc4130f84c69832c1022b7bd7fa5 P f1e775296eb746db8b318f17d26820e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 02:52:48.627803 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_dcdbe8e16ced4132a8b540781fd01d2e [id=1ec97dc2b50a4a3dbbc449559938cead] alter complete (version 1)
I20250112 02:52:48.629206 3310 tablet_bootstrap.cc:492] T 5804fc4130f84c69832c1022b7bd7fa5 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:48.633332 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cabc6fa3f93454aa503dbc5ee58659c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.633694 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1692718d8c964829b0f9f31cfc0101b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.635577 1173 consensus_queue.cc:260] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.638435 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_17388940cd444538a523e34ad3cf51f1" } modify_external_catalogs: true
I20250112 02:52:48.638617 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_17388940cd444538a523e34ad3cf51f1" } modify_external_catalogs: true
I20250112 02:52:48.640020 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1123077650c4e89bffddd6d1264e428" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:48.641379 1173 ts_tablet_manager.cc:1428] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.058s user 0.002s sys 0.000s
W20250112 02:52:48.641590 991 leader_election.cc:343] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.642098 3307 tablet_copy_client.cc:538] T b7340a7a916c4b2fab32e0ba2d0ce322 P f1e775296eb746db8b318f17d26820e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 02:52:48.645392 3307 tablet_bootstrap.cc:492] T b7340a7a916c4b2fab32e0ba2d0ce322 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:48.646035 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_ebbae2e0c50f44fd9e379196c2be90af" } new_table_name: "default.Table_eb7dadebbc494c90996cd0b7d71dc27a" modify_external_catalogs: true
I20250112 02:52:48.646095 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_f72d9efd4f9942ccb28da43e95640497"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:48.646713 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_c0a21e51b1284e2ca2d529c6f24c81fa" } modify_external_catalogs: true
I20250112 02:52:48.646891 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_c0a21e51b1284e2ca2d529c6f24c81fa" } modify_external_catalogs: true
I20250112 02:52:48.645743 1181 tablet_bootstrap.cc:492] T fc2bc28710d047cf92f6e92509c2bbaf P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:48.647218 1181 ts_tablet_manager.cc:1397] T fc2bc28710d047cf92f6e92509c2bbaf P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.234s user 0.003s sys 0.006s
I20250112 02:52:48.648628 1181 raft_consensus.cc:357] T fc2bc28710d047cf92f6e92509c2bbaf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.653952 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_b645d30cd43d4f09a3ac34db205a80f9" } modify_external_catalogs: true
I20250112 02:52:48.654163 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_b645d30cd43d4f09a3ac34db205a80f9" } modify_external_catalogs: true
I20250112 02:52:48.656903 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_b8a99487e7a2452aa61a90f4584e34b4 [id=e63daf724d874c89badafb8d59653f74] alter complete (version 2)
I20250112 02:52:48.657285 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 0925ba2197bb461190b832d062567040 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:49624
W20250112 02:52:48.660844 989 leader_election.cc:343] T ea33053c62dc4707b759d5c1756ab4bd P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: ea33053c62dc4707b759d5c1756ab4bd
I20250112 02:52:48.662178 3273 raft_consensus.cc:491] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:48.662293 3273 raft_consensus.cc:513] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:48.662497 854 leader_election.cc:343] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.662883 3273 leader_election.cc:290] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:48.663169 3347 consensus_queue.cc:1046] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:48.664417 3347 raft_consensus.cc:3049] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Stepping down as leader of term 1
I20250112 02:52:48.664491 3347 raft_consensus.cc:738] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:48.664786 3347 consensus_queue.cc:260] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.665009 3347 raft_consensus.cc:3054] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Advancing to term 2
I20250112 02:52:48.664142 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28228d4200764cc68e83cc0f27b5e3a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.659655 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eed0a9f8354e5391ec2463dafdb683" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:48.664017 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1ae3f34b434ee5bc3519cce8d283a6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:48.666121 855 leader_election.cc:343] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:48.666328 855 leader_election.cc:343] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.649130 1181 raft_consensus.cc:383] T fc2bc28710d047cf92f6e92509c2bbaf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:48.666834 1181 raft_consensus.cc:738] T fc2bc28710d047cf92f6e92509c2bbaf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:48.667626 1181 consensus_queue.cc:260] T fc2bc28710d047cf92f6e92509c2bbaf P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.667961 3110 ts_tablet_manager.cc:1918] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:48.668200 3110 log.cc:1198] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/7b9c845571f44c8eb8753090d63df45f
I20250112 02:52:48.669302 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eed0a9f8354e5391ec2463dafdb683" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:48.662693 854 leader_election.cc:304] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:48.670130 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1ae3f34b434ee5bc3519cce8d283a6" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:48.670538 3397 raft_consensus.cc:2743] T 29d7774d4a284d24b833d50809996fc2 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:48.670666 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acafc4f579534a69a8f307ca4cf72e12" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:48.670898 1181 ts_tablet_manager.cc:1428] T fc2bc28710d047cf92f6e92509c2bbaf P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.024s user 0.000s sys 0.003s
I20250112 02:52:48.671161 3110 ts_tablet_manager.cc:1939] T 7b9c845571f44c8eb8753090d63df45f P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:48.671355 3348 tablet.cc:1721] T 125b12a2f9164653be79d3e6d8313d36 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:48.672248 1181 tablet_bootstrap.cc:492] T be856e4b913e40d58a3f3ba54698cc5f P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:48.672067 854 leader_election.cc:304] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:48.673341 3397 raft_consensus.cc:2743] T a470f84d50df4cf0b4a1e5d248b4c32a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250112 02:52:48.675060 854 leader_election.cc:343] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.676048 3348 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:56348 (request call id 551) took 1028 ms. Trace:
I20250112 02:52:48.676136 3348 rpcz_store.cc:276] 0112 02:52:47.647403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.647509 (+ 106us) service_pool.cc:225] Handling call
0112 02:52:48.676035 (+1028526us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:47.651020 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 02:52:47.651129 (+ 109us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 02:52:47.841068 (+189939us) alter_schema_op.cc:66] Acquired schema lock
0112 02:52:47.841075 (+ 7us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 02:52:47.841102 (+ 27us) alter_schema_op.cc:128] START. Timestamp: 1736650367840985
0112 02:52:47.841105 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:47.841236 (+ 131us) log.cc:844] Serialized 173 byte log entry
0112 02:52:48.668493 (+827257us) op_driver.cc:464] REPLICATION: finished
0112 02:52:48.671332 (+ 2839us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 02:52:48.671812 (+ 480us) log.cc:844] Serialized 5 byte log entry
0112 02:52:48.674527 (+ 2715us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.675827 (+ 1300us) log.cc:844] Serialized 14 byte log entry
0112 02:52:48.676003 (+ 176us) alter_schema_op.cc:73] Released schema lock
0112 02:52:48.676007 (+ 4us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2774,"prepare.queue_time_us":2788,"prepare.run_cpu_time_us":6878,"prepare.run_wall_time_us":190267,"replication_time_us":827357,"thread_start_us":241872,"threads_started":1}]]}
W20250112 02:52:48.677151 441 consensus_peers.cc:487] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e -> Peer dd25f03f25df49178a5c53c636ecf304 (127.0.104.60:39987): Couldn't send request to peer dd25f03f25df49178a5c53c636ecf304. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39987: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250112 02:52:48.675973 854 leader_election.cc:304] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:48.677453 3348 tablet.cc:1721] T 79eb61cedd544c49869471eebe2fd237 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:48.678299 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_b4223b0bf2f348158ec6104d6da3ba42" } new_table_name: "default.Table_1169222985d54d2b843b703bec2578f4" modify_external_catalogs: true
I20250112 02:52:48.679594 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_77741e089cd44e608b680bec456ef0ba" } modify_external_catalogs: true
I20250112 02:52:48.679775 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_77741e089cd44e608b680bec456ef0ba" } modify_external_catalogs: true
I20250112 02:52:48.681680 3397 raft_consensus.cc:2743] T 56ddc025a8314f6f84d229191eb56692 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:48.682277 1173 tablet_bootstrap.cc:492] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:48.682543 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f72d9efd4f9942ccb28da43e95640497 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.678049 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 365) took 2950 ms. Trace:
I20250112 02:52:48.683907 1173 tablet_bootstrap.cc:654] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:48.682927 3110 rpcz_store.cc:276] 0112 02:52:45.727254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.727317 (+ 63us) service_pool.cc:225] Handling call
0112 02:52:48.447346 (+2720029us) ts_tablet_manager.cc:1145] Deleting tablet 7b9c845571f44c8eb8753090d63df45f
0112 02:52:48.447357 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.561810 (+114453us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.667581 (+105771us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.677805 (+ 10224us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":340,"tablet-delete.queue_time_us":2719834}
I20250112 02:52:48.685508 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 7b9c845571f44c8eb8753090d63df45f (table default.Table_7077f9a718664d8b915ee4003502306a [id=6df53b1d0c7a4fa1a18b3202741630b5]) successfully deleted
W20250112 02:52:48.684528 854 leader_election.cc:343] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.686111 854 leader_election.cc:304] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:48.687676 3397 raft_consensus.cc:2743] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:48.688233 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 4a57eb366f594f93a488c8250e195014
I20250112 02:52:48.687886 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 374) took 2692 ms. Trace:
I20250112 02:52:48.688951 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a57eb366f594f93a488c8250e195014 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:48.690471 3111 ts_tablet_manager.cc:1918] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:48.689139 3110 rpcz_store.cc:276] 0112 02:52:45.995758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.001483 (+ 5725us) service_pool.cc:225] Handling call
0112 02:52:48.686508 (+2685025us) ts_tablet_manager.cc:1145] Deleting tablet f60626faf203441ab48c60147ddc3cf8
0112 02:52:48.686518 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.687456 (+ 938us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2633476}
W20250112 02:52:48.691589 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet f60626faf203441ab48c60147ddc3cf8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f60626faf203441ab48c60147ddc3cf8 already in progress: creating tablet
W20250112 02:52:48.692186 854 leader_election.cc:343] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.693094 854 leader_election.cc:304] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:48.693930 1181 tablet_bootstrap.cc:654] T be856e4b913e40d58a3f3ba54698cc5f P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:48.693503 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 375) took 2623 ms. Trace:
I20250112 02:52:48.695268 3397 raft_consensus.cc:2743] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:48.694630 3110 rpcz_store.cc:276] 0112 02:52:46.069583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.069629 (+ 46us) service_pool.cc:225] Handling call
0112 02:52:48.692574 (+2622945us) ts_tablet_manager.cc:1145] Deleting tablet d2028226498b4b41a95c7c836fe03b4d
0112 02:52:48.692585 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.693488 (+ 903us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2622563}
W20250112 02:52:48.697319 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet d2028226498b4b41a95c7c836fe03b4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2028226498b4b41a95c7c836fe03b4d already in progress: creating tablet
I20250112 02:52:48.698738 3110 tablet_replica.cc:331] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
W20250112 02:52:48.698866 854 leader_election.cc:343] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.699704 854 leader_election.cc:304] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:48.701239 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 68539.2us
I20250112 02:52:48.700490 3110 raft_consensus.cc:2238] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:48.701548 3397 raft_consensus.cc:2743] T 7c1ae3f34b434ee5bc3519cce8d283a6 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:48.702775 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 475b8ea7d68b44fba7431a2812c13871 consensus state has no leader
I20250112 02:52:48.703181 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_b9244ba563294ec5ad2ed0579fd5819d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:48.702556 3110 raft_consensus.cc:2267] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:48.704260 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_29a6037a21724b5f9f0e0ee6a598a3dd" } new_table_name: "default.Table_facdca78b120425892df7fd49aad6ea6" modify_external_catalogs: true
I20250112 02:52:48.705674 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_0c2a4844c52549f984bc1a4c77cfd1be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:48.704288 854 leader_election.cc:343] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.709859 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_5bd59377fc6748abaa1fe6436a091cb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:48.712386 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c04f1a2cb04950a9f6170e252c19b4" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:48.712620 771 raft_consensus.cc:2463] T 83c04f1a2cb04950a9f6170e252c19b4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:48.714242 1181 tablet_bootstrap.cc:492] T be856e4b913e40d58a3f3ba54698cc5f P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:48.715237 1181 ts_tablet_manager.cc:1397] T be856e4b913e40d58a3f3ba54698cc5f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.001s
I20250112 02:52:48.715126 3110 ts_tablet_manager.cc:1905] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:48.712411 772 raft_consensus.cc:1235] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [term 2 LEADER]: Rejecting Update request from peer f1e775296eb746db8b318f17d26820e6 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250112 02:52:48.723930 3352 tablet.cc:1721] T 125b12a2f9164653be79d3e6d8313d36 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:48.722358 1181 raft_consensus.cc:357] T be856e4b913e40d58a3f3ba54698cc5f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.724776 1181 raft_consensus.cc:383] T be856e4b913e40d58a3f3ba54698cc5f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:48.725239 1181 raft_consensus.cc:738] T be856e4b913e40d58a3f3ba54698cc5f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:48.726840 448 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 0a5c79bde4a74592b6df3d56578623e9
I20250112 02:52:48.727188 3355 tablet.cc:1721] T cec5346e13164274ae7bc3b6aa241f61 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.727211 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_03c5cd4505234f8b8d3b584d300c0796"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:48.690760 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a57eb366f594f93a488c8250e195014 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:48.727811 3111 log.cc:1198] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/781a9943f705477cb6eae16df4468d2f
I20250112 02:52:48.728087 3111 ts_tablet_manager.cc:1939] T 781a9943f705477cb6eae16df4468d2f P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:48.728344 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 488) took 1846 ms. Trace:
I20250112 02:52:48.728742 3111 rpcz_store.cc:276] 0112 02:52:46.881881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.881925 (+ 44us) service_pool.cc:225] Handling call
0112 02:52:48.528448 (+1646523us) ts_tablet_manager.cc:1145] Deleting tablet 781a9943f705477cb6eae16df4468d2f
0112 02:52:48.528455 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.623031 (+ 94576us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.690391 (+ 67360us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.728332 (+ 37941us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1646388}
I20250112 02:52:48.729288 3111 tablet_replica.cc:331] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:48.729466 3111 raft_consensus.cc:2238] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:48.729585 3111 raft_consensus.cc:2267] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:48.727671 1181 consensus_queue.cc:260] T be856e4b913e40d58a3f3ba54698cc5f P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.731153 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3d291ca3494f05ac23c7fe293e0bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:48.731727 3273 raft_consensus.cc:491] T fc2bc28710d047cf92f6e92509c2bbaf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:48.731859 3273 raft_consensus.cc:513] T fc2bc28710d047cf92f6e92509c2bbaf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.732415 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea33053c62dc4707b759d5c1756ab4bd" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:48.732512 3273 leader_election.cc:290] T fc2bc28710d047cf92f6e92509c2bbaf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:48.694553 3310 tablet_bootstrap.cc:492] T 5804fc4130f84c69832c1022b7bd7fa5 P f1e775296eb746db8b318f17d26820e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 02:52:48.733783 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81bad885cbbf4a8b9a06c8a066f25e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.732657 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acafc4f579534a69a8f307ca4cf72e12" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:48.734334 3310 tablet_bootstrap.cc:492] T 5804fc4130f84c69832c1022b7bd7fa5 P f1e775296eb746db8b318f17d26820e6: Bootstrap complete.
I20250112 02:52:48.734755 3310 ts_tablet_manager.cc:1397] T 5804fc4130f84c69832c1022b7bd7fa5 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.106s user 0.009s sys 0.003s
I20250112 02:52:48.695117 3245 raft_consensus.cc:357] T ca6f080e5fd84084b601b646d4e6c9e2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.736125 3245 raft_consensus.cc:738] T ca6f080e5fd84084b601b646d4e6c9e2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:48.736487 3245 consensus_queue.cc:260] T ca6f080e5fd84084b601b646d4e6c9e2 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.736896 3335 tablet.cc:1721] T 24ebf85f76f44633a20d97964638c0ac P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:48.721410 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99407ead9cf547e3bbe78c0d0b327e6a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:48.738932 3245 ts_tablet_manager.cc:1428] T ca6f080e5fd84084b601b646d4e6c9e2 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.121s user 0.001s sys 0.003s
I20250112 02:52:48.739127 3352 tablet.cc:1721] T e933b58dab0b45e8bf1c49eebb1c53e3 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:48.739674 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99407ead9cf547e3bbe78c0d0b327e6a" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:48.739267 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:48.740967 1181 ts_tablet_manager.cc:1428] T be856e4b913e40d58a3f3ba54698cc5f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20250112 02:52:48.742552 3348 tablet.cc:1721] T cec5346e13164274ae7bc3b6aa241f61 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:52:48.742419 854 leader_election.cc:343] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.743690 1177 tablet_bootstrap.cc:492] T d95f98ae7bdf42518bbff55e257fea88 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:48.745381 3332 tablet.cc:1721] T 8d3a4e76b724452c8b238aae0198e297 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.744629 1181 tablet_bootstrap.cc:492] T 9706187d3f9d4368a8e47962bc73ef3d P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:48.750641 3397 raft_consensus.cc:491] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:48.750761 3397 raft_consensus.cc:513] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
W20250112 02:52:48.751204 855 leader_election.cc:343] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.751302 3397 leader_election.cc:290] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:48.751312 855 leader_election.cc:304] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:48.751782 855 leader_election.cc:343] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.751863 855 leader_election.cc:304] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:48.751984 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2bc28710d047cf92f6e92509c2bbaf" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:48.752259 3273 raft_consensus.cc:2743] T acafc4f579534a69a8f307ca4cf72e12 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:48.752377 3273 raft_consensus.cc:2743] T 99407ead9cf547e3bbe78c0d0b327e6a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:48.758116 3352 tablet.cc:1721] T 8d3a4e76b724452c8b238aae0198e297 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.757555 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:48.758352 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcfa2b316ddd48faa0dc2dd8db8690f9" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:48.760670 1053 raft_consensus.cc:1212] T 164f76edcf8f4bd498dcd2c25fd7477d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
W20250112 02:52:48.762643 854 leader_election.cc:343] T fc2bc28710d047cf92f6e92509c2bbaf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:48.764757 854 leader_election.cc:343] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.765117 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet e933b58dab0b45e8bf1c49eebb1c53e3 (table default.Table_f161f1352b5449ebbac77fcb007d79e0 [id=e184c10af82240f5b1e29fc226269bf0]). Expected version 2 got 1
I20250112 02:52:48.768277 1181 tablet_bootstrap.cc:654] T 9706187d3f9d4368a8e47962bc73ef3d P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:48.773859 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03c5cd4505234f8b8d3b584d300c0796 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.773967 1177 ts_tablet_manager.cc:1397] T d95f98ae7bdf42518bbff55e257fea88 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.336s user 0.014s sys 0.014s
I20250112 02:52:48.775017 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcfa2b316ddd48faa0dc2dd8db8690f9" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:48.729588 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 781a9943f705477cb6eae16df4468d2f (table default.Table_104ccd5f390f47a5878c67ea6de9d554 [id=872482df2ee94056a7d29ce227797f16]) successfully deleted
W20250112 02:52:48.775238 441 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 45668.7us
W20250112 02:52:48.775523 855 leader_election.cc:343] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:48.775573 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9244ba563294ec5ad2ed0579fd5819d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.775614 855 leader_election.cc:304] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:48.775991 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2bc28710d047cf92f6e92509c2bbaf" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:48.776459 855 leader_election.cc:343] T fc2bc28710d047cf92f6e92509c2bbaf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.776572 855 leader_election.cc:304] T fc2bc28710d047cf92f6e92509c2bbaf P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:48.776573 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c2a4844c52549f984bc1a4c77cfd1be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.776782 3000 raft_consensus.cc:2743] T bcfa2b316ddd48faa0dc2dd8db8690f9 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:48.776940 3000 raft_consensus.cc:2743] T fc2bc28710d047cf92f6e92509c2bbaf P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:48.777189 3310 raft_consensus.cc:357] T 5804fc4130f84c69832c1022b7bd7fa5 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.777535 3310 raft_consensus.cc:738] T 5804fc4130f84c69832c1022b7bd7fa5 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:48.777911 3310 consensus_queue.cc:260] T 5804fc4130f84c69832c1022b7bd7fa5 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.778806 3370 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:49624 (request call id 523) took 1696 ms. Trace:
I20250112 02:52:48.778919 3370 rpcz_store.cc:276] 0112 02:52:47.082542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.082677 (+ 135us) service_pool.cc:225] Handling call
0112 02:52:48.778774 (+1696097us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:47.291660 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 02:52:47.291731 (+ 71us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 02:52:47.291739 (+ 8us) alter_schema_op.cc:66] Acquired schema lock
0112 02:52:47.291744 (+ 5us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 02:52:47.291773 (+ 29us) alter_schema_op.cc:128] START. Timestamp: 1736650367291690
0112 02:52:47.291775 (+ 2us) op_driver.cc:348] REPLICATION: starting
0112 02:52:47.322627 (+ 30852us) log.cc:844] Serialized 173 byte log entry
0112 02:52:48.242553 (+919926us) op_driver.cc:464] REPLICATION: finished
0112 02:52:48.365736 (+123183us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 02:52:48.377302 (+ 11566us) log.cc:844] Serialized 5 byte log entry
0112 02:52:48.778515 (+401213us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.778705 (+ 190us) log.cc:844] Serialized 14 byte log entry
0112 02:52:48.778738 (+ 33us) alter_schema_op.cc:73] Released schema lock
0112 02:52:48.778742 (+ 4us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":123140,"prepare.queue_time_us":184594,"prepare.run_cpu_time_us":45073,"prepare.run_wall_time_us":675571,"replication_time_us":950751,"thread_start_us":1104065,"threads_started":2,"wal-append.queue_time_us":638992}]]}
I20250112 02:52:48.779562 938 tablet_copy_service.cc:342] P fe36951a83fb4e57b26e187edbcbfe3d: Request end of tablet copy session f1e775296eb746db8b318f17d26820e6-ca6f080e5fd84084b601b646d4e6c9e2 received from {username='slave'} at 127.0.104.3:52739
I20250112 02:52:48.779660 938 tablet_copy_service.cc:434] P fe36951a83fb4e57b26e187edbcbfe3d: ending tablet copy session f1e775296eb746db8b318f17d26820e6-ca6f080e5fd84084b601b646d4e6c9e2 on tablet ca6f080e5fd84084b601b646d4e6c9e2 with peer f1e775296eb746db8b318f17d26820e6
W20250112 02:52:48.780781 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bd59377fc6748abaa1fe6436a091cb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.782007 3374 tablet.cc:1721] T 125b12a2f9164653be79d3e6d8313d36 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.784633 3307 tablet_bootstrap.cc:492] T b7340a7a916c4b2fab32e0ba2d0ce322 P f1e775296eb746db8b318f17d26820e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 02:52:48.785050 3307 tablet_bootstrap.cc:492] T b7340a7a916c4b2fab32e0ba2d0ce322 P f1e775296eb746db8b318f17d26820e6: Bootstrap complete.
I20250112 02:52:48.785310 3307 ts_tablet_manager.cc:1397] T b7340a7a916c4b2fab32e0ba2d0ce322 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.140s user 0.014s sys 0.000s
I20250112 02:52:48.785706 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet afd6d31155034908ac44a90c3a2ea7d6 (table default.Table_14d8cd331308428f9ce2fff4e64f6394 [id=5cec72cf833347f3bfbca061f8feca41]). Expected version 2 got 1
I20250112 02:52:48.786418 3397 raft_consensus.cc:491] T ca2165362ee541adac724b63f39b1489 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:48.786551 3397 raft_consensus.cc:513] T ca2165362ee541adac724b63f39b1489 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.788075 3397 leader_election.cc:290] T ca2165362ee541adac724b63f39b1489 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:48.788671 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca2165362ee541adac724b63f39b1489" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:48.790269 854 leader_election.cc:343] T ca2165362ee541adac724b63f39b1489 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.790311 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca2165362ee541adac724b63f39b1489" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:48.790624 854 consensus_peers.cc:487] T 09900b42e8a443e680cc35c96a1dbbb7 P fe36951a83fb4e57b26e187edbcbfe3d -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09900b42e8a443e680cc35c96a1dbbb7. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:52:48.790866 855 leader_election.cc:343] T ca2165362ee541adac724b63f39b1489 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.790971 855 leader_election.cc:304] T ca2165362ee541adac724b63f39b1489 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:48.791282 3000 raft_consensus.cc:2743] T ca2165362ee541adac724b63f39b1489 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:48.792533 1181 tablet_bootstrap.cc:492] T 9706187d3f9d4368a8e47962bc73ef3d P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:48.792829 3365 tablet.cc:1721] T e933b58dab0b45e8bf1c49eebb1c53e3 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:48.792938 1181 ts_tablet_manager.cc:1397] T 9706187d3f9d4368a8e47962bc73ef3d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.049s user 0.005s sys 0.000s
I20250112 02:52:48.793642 799 tablet_copy_service.cc:140] P 289afc116aa74408aa5bff74e11c50d8: Received BeginTabletCopySession request for tablet 896908b2726c40f9aabdfc9a6b3283f7 from peer f1e775296eb746db8b318f17d26820e6 ({username='slave'} at 127.0.104.3:52505)
I20250112 02:52:48.793790 799 tablet_copy_service.cc:161] P 289afc116aa74408aa5bff74e11c50d8: Beginning new tablet copy session on tablet 896908b2726c40f9aabdfc9a6b3283f7 from peer f1e775296eb746db8b318f17d26820e6 at {username='slave'} at 127.0.104.3:52505: session id = f1e775296eb746db8b318f17d26820e6-896908b2726c40f9aabdfc9a6b3283f7
I20250112 02:52:48.794308 3266 raft_consensus.cc:491] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:48.794544 3266 raft_consensus.cc:513] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.779431 3370 tablet.cc:1721] T 5a3af0ea4e954053ae89814f3e74e6b7 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.795764 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 5a3af0ea4e954053ae89814f3e74e6b7 (table default.Table_8fd4122d870a475baea2cc3cb97bd01a [id=7d048aa907a74605964b68726ac5d759]). Expected version 1 got 0
I20250112 02:52:48.795934 3111 ts_tablet_manager.cc:1905] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:48.796401 3266 leader_election.cc:290] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:48.796581 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecf730b6985404a8134b497a4b792c8" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:48.796756 909 raft_consensus.cc:2463] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:48.797061 989 leader_election.cc:304] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:48.797483 3266 raft_consensus.cc:2798] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:48.797619 3266 raft_consensus.cc:491] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:48.797757 3266 raft_consensus.cc:3054] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:48.797959 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a470f84d50df4cf0b4a1e5d248b4c32a" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:48.778831 3310 ts_tablet_manager.cc:1428] T 5804fc4130f84c69832c1022b7bd7fa5 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.044s user 0.001s sys 0.001s
I20250112 02:52:48.810752 3110 ts_tablet_manager.cc:1918] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:48.810875 1181 raft_consensus.cc:357] T 9706187d3f9d4368a8e47962bc73ef3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.811975 3110 log.cc:1198] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/37d38f5e525440f8a0f7822620933435
I20250112 02:52:48.812589 1181 raft_consensus.cc:383] T 9706187d3f9d4368a8e47962bc73ef3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:48.813539 1181 raft_consensus.cc:738] T 9706187d3f9d4368a8e47962bc73ef3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:48.816641 1177 raft_consensus.cc:357] T d95f98ae7bdf42518bbff55e257fea88 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.819120 3109 ts_tablet_manager.cc:1918] T e4d42f1611de49d6b6e5aaa9be907054 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:48.820600 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_08a157961c4c4a89a0c891202953f1b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:48.821380 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08a157961c4c4a89a0c891202953f1b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.819394 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecf730b6985404a8134b497a4b792c8" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:48.819543 3371 tablet.cc:1716] T ca6f080e5fd84084b601b646d4e6c9e2 P 289afc116aa74408aa5bff74e11c50d8: Skipping requested alter to schema version 3, tablet already version 3
I20250112 02:52:48.823275 3371 tablet.cc:1721] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:48.816895 1177 raft_consensus.cc:383] T d95f98ae7bdf42518bbff55e257fea88 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:48.823720 1177 raft_consensus.cc:738] T d95f98ae7bdf42518bbff55e257fea88 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:48.823868 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 03276edd7bbe42b5a8ca16c5ccf1b337 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:48.824157 1177 consensus_queue.cc:260] T d95f98ae7bdf42518bbff55e257fea88 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.824860 1177 ts_tablet_manager.cc:1428] T d95f98ae7bdf42518bbff55e257fea88 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.051s user 0.001s sys 0.001s
I20250112 02:52:48.825170 1177 tablet_bootstrap.cc:492] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:48.825188 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_65a72bbe9a9b4ba19f9a1f82ed35bc8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:48.825619 3109 log.cc:1198] T e4d42f1611de49d6b6e5aaa9be907054 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/e4d42f1611de49d6b6e5aaa9be907054
W20250112 02:52:48.825852 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65a72bbe9a9b4ba19f9a1f82ed35bc8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.821425 3307 raft_consensus.cc:357] T b7340a7a916c4b2fab32e0ba2d0ce322 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.829041 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_1a28b756f7994271b098c5404bd23e7a" } new_table_name: "default.Table_3f00edd22a8048609d508e9a9545a361" modify_external_catalogs: true
I20250112 02:52:48.831238 3110 ts_tablet_manager.cc:1939] T 37d38f5e525440f8a0f7822620933435 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:48.832517 3307 raft_consensus.cc:738] T b7340a7a916c4b2fab32e0ba2d0ce322 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:48.831303 1181 consensus_queue.cc:260] T 9706187d3f9d4368a8e47962bc73ef3d P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.837821 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_3912076c1f8f43cda556f9d93ae1ce4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:48.838234 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3912076c1f8f43cda556f9d93ae1ce4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.840485 1181 ts_tablet_manager.cc:1428] T 9706187d3f9d4368a8e47962bc73ef3d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.047s user 0.001s sys 0.001s
I20250112 02:52:48.841087 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_c586400629b14725a93d61c2c6e4ad82" } modify_external_catalogs: true
I20250112 02:52:48.841264 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_c586400629b14725a93d61c2c6e4ad82" } modify_external_catalogs: true
W20250112 02:52:48.842051 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e1ac5b5ea3a6494baf806bd4441e747f has no consensus state
I20250112 02:52:48.843027 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_d7c7a0144de14cb3ba348e747e6ddd0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:48.843423 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7c7a0144de14cb3ba348e747e6ddd0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.844377 1181 tablet_bootstrap.cc:492] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:48.844102 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 376) took 2773 ms. Trace:
I20250112 02:52:48.845561 3109 ts_tablet_manager.cc:1939] T e4d42f1611de49d6b6e5aaa9be907054 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:48.846292 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 477) took 3105 ms. Trace:
I20250112 02:52:48.846442 3109 rpcz_store.cc:276] 0112 02:52:45.740803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.740841 (+ 38us) service_pool.cc:225] Handling call
0112 02:52:48.404485 (+2663644us) ts_tablet_manager.cc:1145] Deleting tablet e4d42f1611de49d6b6e5aaa9be907054
0112 02:52:48.404495 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.717508 (+313013us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.819010 (+101502us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.846262 (+ 27252us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2663420}
I20250112 02:52:48.844906 3110 rpcz_store.cc:276] 0112 02:52:46.070205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.070244 (+ 39us) service_pool.cc:225] Handling call
0112 02:52:48.697888 (+2627644us) ts_tablet_manager.cc:1145] Deleting tablet 37d38f5e525440f8a0f7822620933435
0112 02:52:48.698486 (+ 598us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.762584 (+ 64098us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.810091 (+ 47507us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.843912 (+ 33821us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":538,"tablet-delete.queue_time_us":2627468}
I20250112 02:52:48.847122 3109 tablet_replica.cc:331] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:48.847635 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 37d38f5e525440f8a0f7822620933435 (table default.Table_5aeb2d7bb68c46a589a48526dcc40aaf [id=9d267774bf59428e854651956fbd79be]) successfully deleted
I20250112 02:52:48.848296 3109 raft_consensus.cc:2238] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:48.848942 3109 raft_consensus.cc:2267] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:48.848325 3110 tablet_replica.cc:331] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:48.848304 3307 consensus_queue.cc:260] T b7340a7a916c4b2fab32e0ba2d0ce322 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.849422 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 475b8ea7d68b44fba7431a2812c13871 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:48.851076 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad7b2e22c9584f55b5a4c996bd305e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.852277 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_7a6fda16f2604cf4b5676bd749f16989" } new_table_name: "default.Table_b98f157003f547078e7a86c2190330de" modify_external_catalogs: true
I20250112 02:52:48.853777 799 tablet_copy_source_session.cc:215] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 02:52:48.855779 3307 ts_tablet_manager.cc:1428] T b7340a7a916c4b2fab32e0ba2d0ce322 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.070s user 0.002s sys 0.000s
I20250112 02:52:48.858989 3374 tablet.cc:1721] T 46d1beac4410422bbf1e912d35999f3d P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:48.861863 3110 raft_consensus.cc:2238] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:48.850483 3266 raft_consensus.cc:513] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:48.863442 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5b78e2f2303348dea1ca11c1cfd24e3f consensus state has no leader
I20250112 02:52:48.861900 3000 raft_consensus.cc:491] T 9706187d3f9d4368a8e47962bc73ef3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:48.865674 3348 tablet.cc:1721] T 9b81aee21510452881948e41d7aba9be P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.863159 3110 raft_consensus.cc:2267] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:48.864221 3000 raft_consensus.cc:513] T 9706187d3f9d4368a8e47962bc73ef3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.866623 3110 ts_tablet_manager.cc:1905] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:48.866946 3000 leader_election.cc:290] T 9706187d3f9d4368a8e47962bc73ef3d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:48.865403 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecf730b6985404a8134b497a4b792c8" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:48.867772 912 raft_consensus.cc:3054] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:48.868140 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9706187d3f9d4368a8e47962bc73ef3d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:48.868593 854 leader_election.cc:343] T 9706187d3f9d4368a8e47962bc73ef3d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.868940 3266 leader_election.cc:290] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:48.871678 3355 tablet.cc:1721] T 9b81aee21510452881948e41d7aba9be P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.862802 1181 tablet_bootstrap.cc:654] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:48.879119 3111 ts_tablet_manager.cc:1918] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:48.880049 3110 ts_tablet_manager.cc:1918] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:48.883688 3335 tablet.cc:1716] T 896908b2726c40f9aabdfc9a6b3283f7 P fe36951a83fb4e57b26e187edbcbfe3d: Skipping requested alter to schema version 4, tablet already version 4
I20250112 02:52:48.889285 3110 log.cc:1198] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/cd52261c4a414e7d93a528e2eddfa9db
I20250112 02:52:48.885653 3000 raft_consensus.cc:491] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:48.886049 1181 tablet_bootstrap.cc:492] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:48.889735 1181 ts_tablet_manager.cc:1397] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.046s user 0.001s sys 0.004s
I20250112 02:52:48.889673 3000 raft_consensus.cc:513] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.890214 3000 leader_election.cc:290] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:48.890851 1181 raft_consensus.cc:357] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.891063 1181 raft_consensus.cc:383] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:48.891153 1181 raft_consensus.cc:738] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:48.891361 1181 consensus_queue.cc:260] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:48.892266 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_5bd59377fc6748abaa1fe6436a091cb6" } modify_external_catalogs: true
I20250112 02:52:48.892453 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_5bd59377fc6748abaa1fe6436a091cb6" } modify_external_catalogs: true
I20250112 02:52:48.892509 1181 ts_tablet_manager.cc:1428] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:48.893285 1181 tablet_bootstrap.cc:492] T 574de2480cbb4498b414c284fea30b2c P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:48.894752 1181 tablet_bootstrap.cc:654] T 574de2480cbb4498b414c284fea30b2c P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:48.896463 3110 ts_tablet_manager.cc:1939] T cd52261c4a414e7d93a528e2eddfa9db P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:48.897230 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 377) took 2824 ms. Trace:
I20250112 02:52:48.901360 3110 rpcz_store.cc:276] 0112 02:52:46.073154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.073394 (+ 240us) service_pool.cc:225] Handling call
0112 02:52:48.847977 (+2774583us) ts_tablet_manager.cc:1145] Deleting tablet cd52261c4a414e7d93a528e2eddfa9db
0112 02:52:48.847986 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.875069 (+ 27083us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.879976 (+ 4907us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.897214 (+ 17238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2774182}
I20250112 02:52:48.902565 3110 tablet_replica.cc:331] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:48.902957 3110 raft_consensus.cc:2238] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:48.903154 3110 raft_consensus.cc:2267] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:48.867518 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9706187d3f9d4368a8e47962bc73ef3d" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:48.903947 855 leader_election.cc:343] T 9706187d3f9d4368a8e47962bc73ef3d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.904052 855 leader_election.cc:304] T 9706187d3f9d4368a8e47962bc73ef3d P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:48.904235 3110 ts_tablet_manager.cc:1905] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:48.904516 3000 raft_consensus.cc:2743] T 9706187d3f9d4368a8e47962bc73ef3d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:48.889004 991 leader_election.cc:304] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:48.890529 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b741e4f6abd4695b3ba1d7a667b6046" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:48.905989 3111 log.cc:1198] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/f3771166b77e4f90aaa116ab1a9dc299
I20250112 02:52:48.906298 3111 ts_tablet_manager.cc:1939] T f3771166b77e4f90aaa116ab1a9dc299 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
W20250112 02:52:48.906457 854 leader_election.cc:343] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.872181 3365 tablet.cc:1716] T 896908b2726c40f9aabdfc9a6b3283f7 P 289afc116aa74408aa5bff74e11c50d8: Skipping requested alter to schema version 4, tablet already version 4
I20250112 02:52:48.907624 3365 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:49624 (request call id 524) took 1802 ms. Trace:
I20250112 02:52:48.907749 3365 rpcz_store.cc:276] 0112 02:52:47.105476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.105553 (+ 77us) service_pool.cc:225] Handling call
0112 02:52:48.907610 (+1802057us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:47.964338 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 02:52:47.964432 (+ 94us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 02:52:48.827233 (+862801us) alter_schema_op.cc:66] Acquired schema lock
0112 02:52:48.827241 (+ 8us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 02:52:48.828427 (+ 1186us) alter_schema_op.cc:128] START. Timestamp: 1736650368827708
0112 02:52:48.828430 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 02:52:48.850470 (+ 22040us) log.cc:844] Serialized 173 byte log entry
0112 02:52:48.864980 (+ 14510us) op_driver.cc:464] REPLICATION: finished
0112 02:52:48.872133 (+ 7153us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 02:52:48.906960 (+ 34827us) log.cc:844] Serialized 94 byte log entry
0112 02:52:48.907576 (+ 616us) alter_schema_op.cc:73] Released schema lock
0112 02:52:48.907581 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":7108,"prepare.queue_time_us":848779,"prepare.run_cpu_time_us":44006,"prepare.run_wall_time_us":886497,"raft.queue_time_us":11180,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":230,"replication_time_us":36526}]]}
I20250112 02:52:48.890431 3370 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:49624 (request call id 581) took 1408 ms. Trace:
I20250112 02:52:48.890740 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b741e4f6abd4695b3ba1d7a667b6046" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:48.896102 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecf730b6985404a8134b497a4b792c8" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
W20250112 02:52:48.908800 855 leader_election.cc:343] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.908944 855 leader_election.cc:304] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:48.908849 3370 rpcz_store.cc:276] 0112 02:52:47.470848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.788453 (+317605us) service_pool.cc:225] Handling call
0112 02:52:48.879441 (+1090988us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:52:47.815317 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 02:52:47.877039 (+ 61722us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 02:52:47.877050 (+ 11us) alter_schema_op.cc:66] Acquired schema lock
0112 02:52:47.877059 (+ 9us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 02:52:47.877104 (+ 45us) alter_schema_op.cc:128] START. Timestamp: 1736650367877010
0112 02:52:47.877109 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 02:52:47.877274 (+ 165us) log.cc:844] Serialized 173 byte log entry
0112 02:52:48.406715 (+529441us) op_driver.cc:464] REPLICATION: finished
0112 02:52:48.779399 (+372684us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 02:52:48.854341 (+ 74942us) log.cc:844] Serialized 5 byte log entry
0112 02:52:48.879059 (+ 24718us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.879236 (+ 177us) log.cc:844] Serialized 14 byte log entry
0112 02:52:48.879399 (+ 163us) alter_schema_op.cc:73] Released schema lock
0112 02:52:48.879406 (+ 7us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":372640,"prepare.queue_time_us":44,"prepare.run_cpu_time_us":4629,"prepare.run_wall_time_us":219597,"replication_time_us":529568,"thread_start_us":134881,"threads_started":1,"wal-append.queue_time_us":148657}]]}
I20250112 02:52:48.909267 3000 raft_consensus.cc:2743] T 5b741e4f6abd4695b3ba1d7a667b6046 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:48.906756 3364 raft_consensus.cc:2798] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250112 02:52:48.907508 991 leader_election.cc:343] T ea33053c62dc4707b759d5c1756ab4bd P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.909615 3364 raft_consensus.cc:491] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:48.909646 991 leader_election.cc:304] T ea33053c62dc4707b759d5c1756ab4bd P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:48.909709 3364 raft_consensus.cc:3054] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:48.909837 449 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet fc165cc39111419497f8b5650c85022d
I20250112 02:52:48.909860 3347 raft_consensus.cc:2743] T ea33053c62dc4707b759d5c1756ab4bd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:48.900383 1177 tablet_bootstrap.cc:654] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:48.910614 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_b0278b4907ad40bba09790dee3c75216"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:48.911213 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0278b4907ad40bba09790dee3c75216 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:48.912390 3364 raft_consensus.cc:513] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.913612 3364 leader_election.cc:290] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
W20250112 02:52:48.914084 989 leader_election.cc:343] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 83c04f1a2cb04950a9f6170e252c19b4
W20250112 02:52:48.912393 991 leader_election.cc:343] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.914340 991 leader_election.cc:304] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:48.914623 3364 raft_consensus.cc:2743] T a470f84d50df4cf0b4a1e5d248b4c32a P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250112 02:52:48.916975 991 leader_election.cc:343] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.919306 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 489) took 2037 ms. Trace:
I20250112 02:52:48.919382 3111 rpcz_store.cc:276] 0112 02:52:46.882313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.882359 (+ 46us) service_pool.cc:225] Handling call
0112 02:52:48.729246 (+1846887us) ts_tablet_manager.cc:1145] Deleting tablet f3771166b77e4f90aaa116ab1a9dc299
0112 02:52:48.729255 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.860357 (+131102us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.879047 (+ 18690us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.919294 (+ 40247us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1846610}
I20250112 02:52:48.919528 3282 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 896908b2726c40f9aabdfc9a6b3283f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:48.919730 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet f3771166b77e4f90aaa116ab1a9dc299 (table default.Table_b46fcc60960d4de99ac8b7832afcb4b9 [id=34b5449c89e248189460592f6eae98c6]) successfully deleted
I20250112 02:52:48.923179 3109 ts_tablet_manager.cc:1905] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:48.925622 3282 tablet_copy_client.cc:806] T 896908b2726c40f9aabdfc9a6b3283f7 P f1e775296eb746db8b318f17d26820e6: tablet copy: Starting download of 0 data blocks...
I20250112 02:52:48.925829 3282 tablet_copy_client.cc:670] T 896908b2726c40f9aabdfc9a6b3283f7 P f1e775296eb746db8b318f17d26820e6: tablet copy: Starting download of 1 WAL segments...
W20250112 02:52:48.926829 991 leader_election.cc:343] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:48.929760 799 tablet_copy_service.cc:342] P 289afc116aa74408aa5bff74e11c50d8: Request end of tablet copy session f1e775296eb746db8b318f17d26820e6-5804fc4130f84c69832c1022b7bd7fa5 received from {username='slave'} at 127.0.104.3:52505
I20250112 02:52:48.929776 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c04f1a2cb04950a9f6170e252c19b4" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
I20250112 02:52:48.929912 799 tablet_copy_service.cc:434] P 289afc116aa74408aa5bff74e11c50d8: ending tablet copy session f1e775296eb746db8b318f17d26820e6-5804fc4130f84c69832c1022b7bd7fa5 on tablet 5804fc4130f84c69832c1022b7bd7fa5 with peer f1e775296eb746db8b318f17d26820e6
I20250112 02:52:48.929965 766 raft_consensus.cc:3054] T 83c04f1a2cb04950a9f6170e252c19b4 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:48.930505 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 61eac6b572ff47c185768bee7490edf7 (table default.Table_e8e62bd2fffe4f50849f722dbece2a2c [id=3603f999c77a48dca737bc8ff74eb9de]) successfully deleted
I20250112 02:52:48.932140 766 raft_consensus.cc:2463] T 83c04f1a2cb04950a9f6170e252c19b4 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:48.932649 799 tablet_copy_service.cc:342] P 289afc116aa74408aa5bff74e11c50d8: Request end of tablet copy session f1e775296eb746db8b318f17d26820e6-b7340a7a916c4b2fab32e0ba2d0ce322 received from {username='slave'} at 127.0.104.3:52505
I20250112 02:52:48.932739 799 tablet_copy_service.cc:434] P 289afc116aa74408aa5bff74e11c50d8: ending tablet copy session f1e775296eb746db8b318f17d26820e6-b7340a7a916c4b2fab32e0ba2d0ce322 on tablet b7340a7a916c4b2fab32e0ba2d0ce322 with peer f1e775296eb746db8b318f17d26820e6
I20250112 02:52:48.937450 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 483) took 2084 ms. Trace:
I20250112 02:52:48.937533 3111 rpcz_store.cc:276] 0112 02:52:46.852605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.852667 (+ 62us) service_pool.cc:225] Handling call
0112 02:52:48.937332 (+2084665us) ts_tablet_manager.cc:1145] Deleting tablet b768b7534e034bff86045e8a474bf05d
0112 02:52:48.937342 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.937424 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2049762}
I20250112 02:52:48.937644 3370 tablet.cc:1721] T 125b12a2f9164653be79d3e6d8313d36 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 02:52:48.937964 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet b768b7534e034bff86045e8a474bf05d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b768b7534e034bff86045e8a474bf05d already in progress: creating tablet
I20250112 02:52:48.938376 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_1e8424b3da01469588c376566545694f" } new_table_name: "default.Table_91e8c8533b38479797fb7bcd671451aa" modify_external_catalogs: true
I20250112 02:52:48.940557 3111 tablet_replica.cc:331] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:48.940735 3111 raft_consensus.cc:2238] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:48.940892 3111 raft_consensus.cc:2267] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:48.940248 3371 tablet.cc:1721] T 8d3a4e76b724452c8b238aae0198e297 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.940248 3296 consensus_queue.cc:1035] T 500f0c5c76e54cd8808e6c8611e99d83 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:52:48.942770 3110 ts_tablet_manager.cc:1918] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:48.945217 3110 log.cc:1198] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/b3c5a861a90840d0a33b780ec75bffd2
I20250112 02:52:48.945843 3110 ts_tablet_manager.cc:1939] T b3c5a861a90840d0a33b780ec75bffd2 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
W20250112 02:52:48.949251 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b664fabebd844f2ab744755ceb150511 consensus state has no leader
I20250112 02:52:48.949474 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 68e480204ee94a44a1fef701ef5541f7 from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:48.950028 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_9ecbd17bdc664accbaf672ff5590eeca" } modify_external_catalogs: true
I20250112 02:52:48.950196 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_9ecbd17bdc664accbaf672ff5590eeca" } modify_external_catalogs: true
I20250112 02:52:48.950316 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 384) took 2482 ms. Trace:
I20250112 02:52:48.950511 3110 rpcz_store.cc:276] 0112 02:52:46.465894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.465936 (+ 42us) service_pool.cc:225] Handling call
0112 02:52:48.902004 (+2436068us) ts_tablet_manager.cc:1145] Deleting tablet b3c5a861a90840d0a33b780ec75bffd2
0112 02:52:48.902013 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.934249 (+ 32236us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.942692 (+ 8443us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:48.948256 (+ 5564us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2435923}
I20250112 02:52:48.951793 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet b3c5a861a90840d0a33b780ec75bffd2 (table default.Table_c26869f6cbd540fe980dcf207b5c729d [id=cdd6a2097792484d84bca856f30d7ac1]) successfully deleted
I20250112 02:52:48.952683 3110 tablet_replica.cc:331] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:48.954602 3110 raft_consensus.cc:2238] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:48.955031 3110 raft_consensus.cc:2267] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:48.959112 3110 ts_tablet_manager.cc:1905] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:48.964515 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet e4d42f1611de49d6b6e5aaa9be907054 (table default.Table_f952a0479e8d478bbd9484fc5bf041c8 [id=c01b653cd9fe45b2af721eb97936c5d4]) successfully deleted
I20250112 02:52:48.966012 3310 ts_tablet_manager.cc:927] T 05000c6a48e0450d8e6685ca864f67f0 P f1e775296eb746db8b318f17d26820e6: Initiating tablet copy from peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:48.970288 3111 ts_tablet_manager.cc:1905] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:48.976575 991 leader_election.cc:304] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6; no voters: fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:48.976787 3266 raft_consensus.cc:2798] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:48.976891 3266 raft_consensus.cc:695] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:48.977085 3266 consensus_queue.cc:237] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.977807 3282 tablet_copy_client.cc:538] T 896908b2726c40f9aabdfc9a6b3283f7 P f1e775296eb746db8b318f17d26820e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 02:52:48.982175 3282 tablet_bootstrap.cc:492] T 896908b2726c40f9aabdfc9a6b3283f7 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:48.989809 3348 tablet.cc:1721] T ead501d76bb1412e9839fb36b9911e6f P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250112 02:52:48.992221 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet b664fabebd844f2ab744755ceb150511 (table default.Table_91e8c8533b38479797fb7bcd671451aa [id=3501de8bb4f04582af6b7029ab52c9b0], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet b664fabebd844f2ab744755ceb150511 consensus state has no leader
I20250112 02:52:48.995282 3370 tablet.cc:1721] T 9b81aee21510452881948e41d7aba9be P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:48.996446 1177 tablet_bootstrap.cc:492] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:48.996623 1177 ts_tablet_manager.cc:1397] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.172s user 0.005s sys 0.001s
I20250112 02:52:48.997380 1177 raft_consensus.cc:357] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.997663 1177 raft_consensus.cc:383] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:48.997754 1177 raft_consensus.cc:738] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:48.998041 1177 consensus_queue.cc:260] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:48.998857 3310 tablet_copy_client.cc:323] T 05000c6a48e0450d8e6685ca864f67f0 P f1e775296eb746db8b318f17d26820e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.104.1:35007
I20250112 02:52:48.999984 3347 raft_consensus.cc:491] T 93aba9b4d1dd485cb62c4827c916a1d5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.000094 3347 raft_consensus.cc:513] T 93aba9b4d1dd485cb62c4827c916a1d5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.000572 3347 leader_election.cc:290] T 93aba9b4d1dd485cb62c4827c916a1d5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:49.000806 3352 tablet.cc:1721] T b9d525dae23744a4adfc2f58cb9945ce P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:52:49.001358 989 leader_election.cc:343] T 93aba9b4d1dd485cb62c4827c916a1d5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: 93aba9b4d1dd485cb62c4827c916a1d5
I20250112 02:52:49.001495 3374 tablet.cc:1721] T b9d525dae23744a4adfc2f58cb9945ce P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:49.002135 799 tablet_copy_service.cc:140] P 289afc116aa74408aa5bff74e11c50d8: Received BeginTabletCopySession request for tablet 05000c6a48e0450d8e6685ca864f67f0 from peer f1e775296eb746db8b318f17d26820e6 ({username='slave'} at 127.0.104.3:52505)
I20250112 02:52:49.002241 799 tablet_copy_service.cc:161] P 289afc116aa74408aa5bff74e11c50d8: Beginning new tablet copy session on tablet 05000c6a48e0450d8e6685ca864f67f0 from peer f1e775296eb746db8b318f17d26820e6 at {username='slave'} at 127.0.104.3:52505: session id = f1e775296eb746db8b318f17d26820e6-05000c6a48e0450d8e6685ca864f67f0
I20250112 02:52:49.002637 3296 raft_consensus.cc:491] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.002753 3296 raft_consensus.cc:513] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.003063 3348 tablet.cc:1721] T e933b58dab0b45e8bf1c49eebb1c53e3 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:49.003672 904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3499374810fb412e938325434c35618d" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:49.003890 904 raft_consensus.cc:2463] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:49.003969 3332 tablet.cc:1721] T 5a3af0ea4e954053ae89814f3e74e6b7 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:49.004246 713 leader_election.cc:304] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:49.004484 3106 raft_consensus.cc:2798] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:49.004554 3106 raft_consensus.cc:491] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:49.004606 3106 raft_consensus.cc:3054] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:49.004719 799 tablet_copy_source_session.cc:215] T 05000c6a48e0450d8e6685ca864f67f0 P 289afc116aa74408aa5bff74e11c50d8: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 02:52:49.004860 3296 leader_election.cc:290] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:49.006510 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93aba9b4d1dd485cb62c4827c916a1d5" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:49.006747 3106 raft_consensus.cc:513] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.007767 904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3499374810fb412e938325434c35618d" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:49.007984 904 raft_consensus.cc:3054] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:49.009090 991 leader_election.cc:343] T 93aba9b4d1dd485cb62c4827c916a1d5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.009179 991 leader_election.cc:304] T 93aba9b4d1dd485cb62c4827c916a1d5 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1e775296eb746db8b318f17d26820e6; no voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:49.009419 3347 raft_consensus.cc:2743] T 93aba9b4d1dd485cb62c4827c916a1d5 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:49.011315 904 raft_consensus.cc:2463] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:49.011929 713 leader_election.cc:304] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:49.012437 3318 raft_consensus.cc:2798] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:49.012570 3318 raft_consensus.cc:695] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:49.012787 3318 consensus_queue.cc:237] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.016005 3106 leader_election.cc:290] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:49.017392 3110 ts_tablet_manager.cc:1918] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:49.017504 3110 log.cc:1198] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/5804fc4130f84c69832c1022b7bd7fa5
I20250112 02:52:49.017951 3110 ts_tablet_manager.cc:1939] T 5804fc4130f84c69832c1022b7bd7fa5 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:49.019886 3310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05000c6a48e0450d8e6685ca864f67f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.019637 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 382) took 2553 ms. Trace:
I20250112 02:52:49.020336 3110 rpcz_store.cc:276] 0112 02:52:46.465738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.466252 (+ 514us) service_pool.cc:225] Handling call
0112 02:52:48.952640 (+2486388us) ts_tablet_manager.cc:1145] Deleting tablet 5804fc4130f84c69832c1022b7bd7fa5
0112 02:52:48.952650 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.967446 (+ 14796us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.017309 (+ 49863us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.019621 (+ 2312us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2486254}
I20250112 02:52:49.020713 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]) successfully deleted
I20250112 02:52:49.022572 3109 ts_tablet_manager.cc:1918] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 02:52:49.023797 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 385) took 2557 ms. Trace:
I20250112 02:52:49.023880 3110 rpcz_store.cc:276] 0112 02:52:46.465980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.466415 (+ 435us) service_pool.cc:225] Handling call
0112 02:52:49.020602 (+2554187us) ts_tablet_manager.cc:1145] Deleting tablet 61d7703b15024fe7a35805345d5e81ab
0112 02:52:49.023682 (+ 3080us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.023782 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2554073}
W20250112 02:52:49.024263 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 61d7703b15024fe7a35805345d5e81ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61d7703b15024fe7a35805345d5e81ab already in progress: creating tablet
I20250112 02:52:49.024966 3110 tablet_replica.cc:331] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:49.025161 3110 raft_consensus.cc:2238] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:49.025288 3110 raft_consensus.cc:2267] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:49.025727 3110 ts_tablet_manager.cc:1905] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:49.030349 3355 tablet.cc:1721] T afd6d31155034908ac44a90c3a2ea7d6 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 02:52:49.035979 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 113152us
I20250112 02:52:49.035681 3282 tablet.cc:1721] T 896908b2726c40f9aabdfc9a6b3283f7 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 02:52:49.038264 3282 tablet_metadata.cc:623] Not flushing: waiting for 1 pins to be released.
I20250112 02:52:49.038946 3282 tablet_bootstrap.cc:492] T 896908b2726c40f9aabdfc9a6b3283f7 P f1e775296eb746db8b318f17d26820e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250112 02:52:49.039341 3282 tablet_bootstrap.cc:492] T 896908b2726c40f9aabdfc9a6b3283f7 P f1e775296eb746db8b318f17d26820e6: Bootstrap complete.
I20250112 02:52:49.044431 3110 ts_tablet_manager.cc:1918] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:49.044538 3110 log.cc:1198] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/3499374810fb412e938325434c35618d
I20250112 02:52:49.044849 3110 ts_tablet_manager.cc:1939] T 3499374810fb412e938325434c35618d P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:49.045550 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 383) took 2579 ms. Trace:
I20250112 02:52:49.045637 3110 rpcz_store.cc:276] 0112 02:52:46.465823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.468365 (+ 2542us) service_pool.cc:225] Handling call
0112 02:52:49.024911 (+2556546us) ts_tablet_manager.cc:1145] Deleting tablet 3499374810fb412e938325434c35618d
0112 02:52:49.024931 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.028339 (+ 3408us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.044352 (+ 16013us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.045534 (+ 1182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2555609}
I20250112 02:52:49.045930 3111 ts_tablet_manager.cc:1918] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:49.046031 3111 log.cc:1198] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/a8e7882e07784bdda18dc1ffd20963c8
I20250112 02:52:49.046020 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 3499374810fb412e938325434c35618d (table default.Table_7870d93893b74b4b96d1a3809a9015e6 [id=4355589b5ccc4162ab8700e31173d1e5]) successfully deleted
I20250112 02:52:49.046314 3111 ts_tablet_manager.cc:1939] T a8e7882e07784bdda18dc1ffd20963c8 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:49.048087 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 386) took 2553 ms. Trace:
I20250112 02:52:49.048909 3000 raft_consensus.cc:491] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.049475 3000 raft_consensus.cc:513] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.049418 3110 rpcz_store.cc:276] 0112 02:52:46.494215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.494291 (+ 76us) service_pool.cc:225] Handling call
0112 02:52:49.048004 (+2553713us) ts_tablet_manager.cc:1145] Deleting tablet ea33053c62dc4707b759d5c1756ab4bd
0112 02:52:49.048012 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.048072 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2553537}
I20250112 02:52:49.050019 3000 leader_election.cc:290] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:49.050576 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet ea33053c62dc4707b759d5c1756ab4bd because the tablet was not found. No further retry: Not found: Tablet not found: ea33053c62dc4707b759d5c1756ab4bd
I20250112 02:52:49.050428 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 387) took 2556 ms. Trace:
I20250112 02:52:49.051055 3110 rpcz_store.cc:276] 0112 02:52:46.494380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.494535 (+ 155us) service_pool.cc:225] Handling call
0112 02:52:49.050070 (+2555535us) ts_tablet_manager.cc:1145] Deleting tablet 76009bac22ce4136b148b493a004a0f7
0112 02:52:49.050078 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.050414 (+ 336us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2555410}
I20250112 02:52:49.051381 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 7320ac55dbfa47fe851fff94d32e1ffe (table default.Table_ef71f75b40b14fa5b96ee12db588b5d8 [id=8cc378b0da6e4b77bc2c6822cd0844fe]). Expected version 1 got 0
W20250112 02:52:49.051651 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 76009bac22ce4136b148b493a004a0f7 because the tablet was not found. No further retry: Not found: Tablet not found: 76009bac22ce4136b148b493a004a0f7
I20250112 02:52:49.051775 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 125b12a2f9164653be79d3e6d8313d36 (table default.Table_d9117b8cf65240dab1d9c1f61fe05192 [id=56974bbf7ff24cf0b2e9cf834e1c99f0]). Expected version 2 got 0
I20250112 02:52:49.052000 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:49.052356 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 896908b2726c40f9aabdfc9a6b3283f7 (table default.Table_67b6f55a9ba84491bdda41cdf82c25fc [id=63c20c66463d4a30b7e1558fecce2704]). Expected version 4 got 0
I20250112 02:52:49.052452 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5524d75599f04becbe6a94426adb7026" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:49.052673 451 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_a9907ed8bfe14625977ec98b67e9d599 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 2 got 0
W20250112 02:52:49.052877 854 leader_election.cc:343] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.055107 1181 tablet_bootstrap.cc:492] T 574de2480cbb4498b414c284fea30b2c P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:49.055267 1181 ts_tablet_manager.cc:1397] T 574de2480cbb4498b414c284fea30b2c P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.163s user 0.012s sys 0.015s
I20250112 02:52:49.055912 1181 raft_consensus.cc:357] T 574de2480cbb4498b414c284fea30b2c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.056177 1181 raft_consensus.cc:383] T 574de2480cbb4498b414c284fea30b2c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.056280 1181 raft_consensus.cc:738] T 574de2480cbb4498b414c284fea30b2c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.056517 1181 consensus_queue.cc:260] T 574de2480cbb4498b414c284fea30b2c P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.057047 1181 ts_tablet_manager.cc:1428] T 574de2480cbb4498b414c284fea30b2c P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:49.057228 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 388) took 2560 ms. Trace:
I20250112 02:52:49.057495 1181 tablet_bootstrap.cc:492] T 64b0d58af4cf4adcbeb3ef5070f2982d P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:49.057548 3110 rpcz_store.cc:276] 0112 02:52:46.496704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.497649 (+ 945us) service_pool.cc:225] Handling call
0112 02:52:49.051971 (+2554322us) ts_tablet_manager.cc:1145] Deleting tablet cbe30f9860e2492b90e8cedf8ff9daba
0112 02:52:49.057153 (+ 5182us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.057216 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2554150}
W20250112 02:52:49.059736 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet cbe30f9860e2492b90e8cedf8ff9daba because the tablet was not found. No further retry: Not found: Tablet not found: cbe30f9860e2492b90e8cedf8ff9daba
I20250112 02:52:49.060967 1181 tablet_bootstrap.cc:654] T 64b0d58af4cf4adcbeb3ef5070f2982d P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.061167 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 389) took 2563 ms. Trace:
I20250112 02:52:49.063580 3110 rpcz_store.cc:276] 0112 02:52:46.497434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.498606 (+ 1172us) service_pool.cc:225] Handling call
0112 02:52:49.061099 (+2562493us) ts_tablet_manager.cc:1145] Deleting tablet 3d662e0ce13f447595e5f92b94de3018
0112 02:52:49.061107 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.061154 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2562369}
W20250112 02:52:49.064033 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 3d662e0ce13f447595e5f92b94de3018 because the tablet was not found. No further retry: Not found: Tablet not found: 3d662e0ce13f447595e5f92b94de3018
I20250112 02:52:49.064339 3110 tablet_replica.cc:331] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:49.064559 3110 raft_consensus.cc:2238] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:49.064759 3110 raft_consensus.cc:2267] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:49.065264 3110 ts_tablet_manager.cc:1905] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:49.069449 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 03276edd7bbe42b5a8ca16c5ccf1b337
I20250112 02:52:49.069890 455 catalog_manager.cc:6596] ReplaceTablet: tablet 03276edd7bbe42b5a8ca16c5ccf1b337 deleted and replaced by tablet dca0a8184de04e16ab57125444820dec
I20250112 02:52:49.070092 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 9bbef42e11a843b18ed582ab4d04a92a from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:49.070091 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 03276edd7bbe42b5a8ca16c5ccf1b337 with delete_type TABLET_DATA_DELETED (replaced by tablet dca0a8184de04e16ab57125444820dec) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:49.070659 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 03276edd7bbe42b5a8ca16c5ccf1b337 with delete_type TABLET_DATA_DELETED (replaced by tablet dca0a8184de04e16ab57125444820dec) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:49.075059 1173 tablet_bootstrap.cc:492] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:49.075168 1173 ts_tablet_manager.cc:1397] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.393s user 0.018s sys 0.000s
I20250112 02:52:49.075685 1173 raft_consensus.cc:357] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.075793 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 430a8652c53e4801a1c72de529c2adcb
I20250112 02:52:49.075848 1173 raft_consensus.cc:383] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.075925 1173 raft_consensus.cc:738] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.076129 1173 consensus_queue.cc:260] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.076503 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 430a8652c53e4801a1c72de529c2adcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:49.076560 1173 ts_tablet_manager.cc:1428] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 02:52:49.076537 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_6abbbb59fff943f2ad3f7c778428db4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:49.076683 3109 log.cc:1198] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/09d06e2e8c13464db8cf801a292bdc99
W20250112 02:52:49.077018 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6abbbb59fff943f2ad3f7c778428db4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.077030 3109 ts_tablet_manager.cc:1939] T 09d06e2e8c13464db8cf801a292bdc99 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:49.077679 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 478) took 3319 ms. Trace:
I20250112 02:52:49.077749 3109 rpcz_store.cc:276] 0112 02:52:45.758679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.758726 (+ 47us) service_pool.cc:225] Handling call
0112 02:52:48.847001 (+3088275us) ts_tablet_manager.cc:1145] Deleting tablet 09d06e2e8c13464db8cf801a292bdc99
0112 02:52:48.847011 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.019876 (+172865us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.022502 (+ 2626us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.077662 (+ 55160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3088023}
I20250112 02:52:49.078016 3109 tablet_replica.cc:331] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:49.078177 3109 raft_consensus.cc:2238] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:49.078178 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 430a8652c53e4801a1c72de529c2adcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:49.025002 3133 raft_consensus.cc:491] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.078385 3109 raft_consensus.cc:2267] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:49.078413 3133 raft_consensus.cc:513] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.078790 3109 ts_tablet_manager.cc:1905] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:49.079131 3133 leader_election.cc:290] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:49.081223 3110 ts_tablet_manager.cc:1918] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:49.081272 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 475b8ea7d68b44fba7431a2812c13871
I20250112 02:52:49.081310 3110 log.cc:1198] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/681a930541a84f0197ecafe670278e70
I20250112 02:52:49.081544 3110 ts_tablet_manager.cc:1939] T 681a930541a84f0197ecafe670278e70 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:49.081688 450 catalog_manager.cc:6596] ReplaceTablet: tablet 475b8ea7d68b44fba7431a2812c13871 deleted and replaced by tablet 0048bafbc21e4905864aa40f5daa2d7b
I20250112 02:52:49.081743 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 475b8ea7d68b44fba7431a2812c13871 with delete_type TABLET_DATA_DELETED (replaced by tablet 0048bafbc21e4905864aa40f5daa2d7b) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:49.081933 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_69f363c4da234792886b544823b3214e" } new_table_name: "default.Table_58c94c853e5e462b8dc766083a429575" modify_external_catalogs: true
I20250112 02:52:49.082684 1181 tablet_bootstrap.cc:492] T 64b0d58af4cf4adcbeb3ef5070f2982d P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:49.082792 1181 ts_tablet_manager.cc:1397] T 64b0d58af4cf4adcbeb3ef5070f2982d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20250112 02:52:49.083034 3000 raft_consensus.cc:491] T 3dab0466476c4a9aa265271c1abd91a4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.083145 3000 raft_consensus.cc:513] T 3dab0466476c4a9aa265271c1abd91a4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.083348 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 475b8ea7d68b44fba7431a2812c13871 with delete_type TABLET_DATA_DELETED (replaced by tablet 0048bafbc21e4905864aa40f5daa2d7b) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:49.083707 3000 leader_election.cc:290] T 3dab0466476c4a9aa265271c1abd91a4 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:49.083990 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dab0466476c4a9aa265271c1abd91a4" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:49.084414 855 leader_election.cc:343] T 3dab0466476c4a9aa265271c1abd91a4 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.084885 1181 raft_consensus.cc:357] T 64b0d58af4cf4adcbeb3ef5070f2982d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.085098 1181 raft_consensus.cc:383] T 64b0d58af4cf4adcbeb3ef5070f2982d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.085175 1181 raft_consensus.cc:738] T 64b0d58af4cf4adcbeb3ef5070f2982d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.085392 1181 consensus_queue.cc:260] T 64b0d58af4cf4adcbeb3ef5070f2982d P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.086109 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dab0466476c4a9aa265271c1abd91a4" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:49.086475 854 leader_election.cc:343] T 3dab0466476c4a9aa265271c1abd91a4 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.086552 854 leader_election.cc:304] T 3dab0466476c4a9aa265271c1abd91a4 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:49.086727 3000 raft_consensus.cc:2743] T 3dab0466476c4a9aa265271c1abd91a4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:49.086719 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 490) took 2203 ms. Trace:
I20250112 02:52:49.086834 1181 ts_tablet_manager.cc:1428] T 64b0d58af4cf4adcbeb3ef5070f2982d P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:49.087107 1181 tablet_bootstrap.cc:492] T f60626faf203441ab48c60147ddc3cf8 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:49.089172 1181 tablet_bootstrap.cc:654] T f60626faf203441ab48c60147ddc3cf8 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.026079 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a57eb366f594f93a488c8250e195014 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:49.046857 1177 ts_tablet_manager.cc:1428] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.050s user 0.001s sys 0.001s
I20250112 02:52:49.091068 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 390) took 2575 ms. Trace:
I20250112 02:52:49.091111 1177 tablet_bootstrap.cc:492] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:49.091157 3110 rpcz_store.cc:276] 0112 02:52:46.515744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.515794 (+ 50us) service_pool.cc:225] Handling call
0112 02:52:49.064298 (+2548504us) ts_tablet_manager.cc:1145] Deleting tablet 681a930541a84f0197ecafe670278e70
0112 02:52:49.064307 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.072026 (+ 7719us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.081154 (+ 9128us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.091051 (+ 9897us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2548338}
I20250112 02:52:49.091677 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 681a930541a84f0197ecafe670278e70 (table default.Table_8293882503214cf284e12ff8cc75b9c7 [id=15f8694f2e1a403fbf934065a1a79e17]) successfully deleted
I20250112 02:52:49.086791 3111 rpcz_store.cc:276] 0112 02:52:46.882800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.882865 (+ 65us) service_pool.cc:225] Handling call
0112 02:52:48.940519 (+2057654us) ts_tablet_manager.cc:1145] Deleting tablet a8e7882e07784bdda18dc1ffd20963c8
0112 02:52:48.940527 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:48.983087 (+ 42560us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.045848 (+ 62761us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.086704 (+ 40856us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36,"spinlock_wait_cycles":9600,"tablet-delete.queue_time_us":2052951}
I20250112 02:52:49.092244 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet a8e7882e07784bdda18dc1ffd20963c8 (table default.Table_f3695b00a89b4353903587a6c1418d8c [id=d1781738b8a74f139b09863918cbb21d]) successfully deleted
I20250112 02:52:49.093060 1177 tablet_bootstrap.cc:654] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.093701 912 raft_consensus.cc:2463] T 2ecf730b6985404a8134b497a4b792c8 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:49.074316 3374 tablet.cc:1721] T cec5346e13164274ae7bc3b6aa241f61 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:49.023022 3371 tablet.cc:1721] T afd6d31155034908ac44a90c3a2ea7d6 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:49.053292 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5524d75599f04becbe6a94426adb7026" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:49.098155 1177 tablet_bootstrap.cc:492] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:49.098296 1177 ts_tablet_manager.cc:1397] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20250112 02:52:49.098910 1177 raft_consensus.cc:357] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.100684 1177 raft_consensus.cc:383] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.100785 1177 raft_consensus.cc:738] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.101048 1177 consensus_queue.cc:260] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.101536 1177 ts_tablet_manager.cc:1428] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 02:52:49.101904 1177 tablet_bootstrap.cc:492] T 3d418cda9ed5449fa45690130cd6d5bc P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:49.104728 1177 tablet_bootstrap.cc:654] T 3d418cda9ed5449fa45690130cd6d5bc P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:49.100821 714 leader_election.cc:343] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 3499374810fb412e938325434c35618d
W20250112 02:52:49.106141 714 leader_election.cc:343] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: 3499374810fb412e938325434c35618d
I20250112 02:52:49.106175 1019 tablet_service.cc:1467] Processing CreateTablet for tablet c69e38fcc8bb4868a5c5830edcafad6a (DEFAULT_TABLE table=default.Table_d9871dbfe3f84ae88ade407167ea927e [id=220fbde75b4b425e84bb77efc68d1fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.107789 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 09d06e2e8c13464db8cf801a292bdc99 (table default.Table_2c52fead633649dd869763e5aa532be7 [id=e3bb489c53454222a79702b798c0c2f1]) successfully deleted
I20250112 02:52:49.109826 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 499) took 2187 ms. Trace:
I20250112 02:52:49.109910 3111 rpcz_store.cc:276] 0112 02:52:46.922777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.925063 (+ 2286us) service_pool.cc:225] Handling call
0112 02:52:49.092450 (+2167387us) ts_tablet_manager.cc:1145] Deleting tablet a7db121c542f423bbffd2e36c459c3f0
0112 02:52:49.109473 (+ 17023us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.109811 (+ 338us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2167140}
W20250112 02:52:49.110471 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet a7db121c542f423bbffd2e36c459c3f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7db121c542f423bbffd2e36c459c3f0 already in progress: creating tablet
I20250112 02:52:49.110706 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69e38fcc8bb4868a5c5830edcafad6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.112679 3282 ts_tablet_manager.cc:1397] T 896908b2726c40f9aabdfc9a6b3283f7 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.131s user 0.013s sys 0.004s
I20250112 02:52:49.117249 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_abd6c7fd7997466eb33bfee729fd3146"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:49.117762 1177 tablet_bootstrap.cc:492] T 3d418cda9ed5449fa45690130cd6d5bc P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
W20250112 02:52:49.117739 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abd6c7fd7997466eb33bfee729fd3146 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.117856 1177 ts_tablet_manager.cc:1397] T 3d418cda9ed5449fa45690130cd6d5bc P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.001s
I20250112 02:52:49.118470 1177 raft_consensus.cc:357] T 3d418cda9ed5449fa45690130cd6d5bc P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.118624 1177 raft_consensus.cc:383] T 3d418cda9ed5449fa45690130cd6d5bc P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.118675 1177 raft_consensus.cc:738] T 3d418cda9ed5449fa45690130cd6d5bc P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.118876 1177 consensus_queue.cc:260] T 3d418cda9ed5449fa45690130cd6d5bc P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.119457 1177 ts_tablet_manager.cc:1428] T 3d418cda9ed5449fa45690130cd6d5bc P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:49.119761 1177 tablet_bootstrap.cc:492] T f3771166b77e4f90aaa116ab1a9dc299 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:49.121356 1177 tablet_bootstrap.cc:654] T f3771166b77e4f90aaa116ab1a9dc299 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.095599 3110 tablet_replica.cc:331] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:49.123544 3110 raft_consensus.cc:2238] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:49.123713 3110 raft_consensus.cc:2267] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:49.124210 3110 ts_tablet_manager.cc:1905] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:49.124980 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 03276edd7bbe42b5a8ca16c5ccf1b337 with delete_type TABLET_DATA_DELETED (replaced by tablet dca0a8184de04e16ab57125444820dec) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:49.125808 447 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 24ebf85f76f44633a20d97964638c0ac (table default.Table_a9907ed8bfe14625977ec98b67e9d599 [id=386e3565dbf2490997e5cc81d6f02567]). Expected version 2 got 0
I20250112 02:52:49.126411 447 catalog_manager.cc:5618] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451) does not have the latest schema for tablet 3dab0466476c4a9aa265271c1abd91a4 (table default.Table_74701c1528924a7b840a89c1dd7eda96 [id=e34a839b388240f0afba216f60d56d24]). Expected version 1 got 0
I20250112 02:52:49.096599 1181 tablet_bootstrap.cc:492] T f60626faf203441ab48c60147ddc3cf8 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:49.127276 1181 ts_tablet_manager.cc:1397] T f60626faf203441ab48c60147ddc3cf8 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.000s
I20250112 02:52:49.127902 1181 raft_consensus.cc:357] T f60626faf203441ab48c60147ddc3cf8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.128123 1181 raft_consensus.cc:383] T f60626faf203441ab48c60147ddc3cf8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.128172 1181 raft_consensus.cc:738] T f60626faf203441ab48c60147ddc3cf8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.128386 1181 consensus_queue.cc:260] T f60626faf203441ab48c60147ddc3cf8 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.128926 1181 ts_tablet_manager.cc:1428] T f60626faf203441ab48c60147ddc3cf8 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:49.129217 1181 tablet_bootstrap.cc:492] T 1164804f2e2644b589ff727f499ed6a1 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:49.097297 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d47f6d663934b25a0200ee9fa81165d" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
W20250112 02:52:49.097775 855 leader_election.cc:343] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.130448 855 leader_election.cc:304] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:49.130981 3397 raft_consensus.cc:2743] T 5524d75599f04becbe6a94426adb7026 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:49.131186 1181 tablet_bootstrap.cc:654] T 1164804f2e2644b589ff727f499ed6a1 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.131639 895 tablet_service.cc:1467] Processing CreateTablet for tablet c69e38fcc8bb4868a5c5830edcafad6a (DEFAULT_TABLE table=default.Table_d9871dbfe3f84ae88ade407167ea927e [id=220fbde75b4b425e84bb77efc68d1fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.132035 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69e38fcc8bb4868a5c5830edcafad6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.134450 3110 ts_tablet_manager.cc:1918] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:49.134534 3110 log.cc:1198] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/4cfe1a1595014c27bbd350cec169d71a
I20250112 02:52:49.134825 3110 ts_tablet_manager.cc:1939] T 4cfe1a1595014c27bbd350cec169d71a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:49.136090 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 391) took 2620 ms. Trace:
I20250112 02:52:49.136540 3110 rpcz_store.cc:276] 0112 02:52:46.515889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.515991 (+ 102us) service_pool.cc:225] Handling call
0112 02:52:49.095559 (+2579568us) ts_tablet_manager.cc:1145] Deleting tablet 4cfe1a1595014c27bbd350cec169d71a
0112 02:52:49.095568 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.127058 (+ 31490us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.134382 (+ 7324us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.136075 (+ 1693us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2579464}
I20250112 02:52:49.138247 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet 4cfe1a1595014c27bbd350cec169d71a (table default.Table_93133ac10ebe44d58e69c21d228c1c5e [id=d8ecda6d109a47c494582f7a77cd92f4]) successfully deleted
I20250112 02:52:49.139150 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 392) took 2623 ms. Trace:
I20250112 02:52:49.139233 3110 rpcz_store.cc:276] 0112 02:52:46.515984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.516183 (+ 199us) service_pool.cc:225] Handling call
0112 02:52:49.136907 (+2620724us) ts_tablet_manager.cc:1145] Deleting tablet a4328137407843e1879493ef0c56ffc8
0112 02:52:49.139074 (+ 2167us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.139137 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2620622}
I20250112 02:52:49.139549 3110 tablet_replica.cc:331] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
W20250112 02:52:49.139598 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet a4328137407843e1879493ef0c56ffc8 because the tablet was not found. No further retry: Not found: Tablet not found: a4328137407843e1879493ef0c56ffc8
I20250112 02:52:49.139737 3110 raft_consensus.cc:2238] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:49.139848 3110 raft_consensus.cc:2267] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:49.139986 909 raft_consensus.cc:2463] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:49.140255 3110 ts_tablet_manager.cc:1905] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:49.140333 713 leader_election.cc:304] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:49.140635 3425 raft_consensus.cc:2798] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:49.140693 3425 raft_consensus.cc:491] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:49.140738 3425 raft_consensus.cc:3054] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:49.143193 3425 raft_consensus.cc:513] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.144145 3425 leader_election.cc:290] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:49.144402 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d47f6d663934b25a0200ee9fa81165d" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:49.144590 909 raft_consensus.cc:3054] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:49.146854 909 raft_consensus.cc:2463] T 2d47f6d663934b25a0200ee9fa81165d P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:49.147184 713 leader_election.cc:304] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:49.147516 3425 raft_consensus.cc:2798] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:49.147651 3425 raft_consensus.cc:695] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:49.147927 3425 consensus_queue.cc:237] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.151434 3109 ts_tablet_manager.cc:1918] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:49.153244 3109 log.cc:1198] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/7b9c845571f44c8eb8753090d63df45f
I20250112 02:52:49.153669 3109 ts_tablet_manager.cc:1939] T 7b9c845571f44c8eb8753090d63df45f P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:49.154420 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 479) took 3395 ms. Trace:
I20250112 02:52:49.159312 3000 raft_consensus.cc:491] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.159423 3000 raft_consensus.cc:513] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.159704 3110 ts_tablet_manager.cc:1918] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:49.159785 3110 log.cc:1198] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/d0cb54003a83472cb9c57ace3e65b0be
I20250112 02:52:49.160027 3110 ts_tablet_manager.cc:1939] T d0cb54003a83472cb9c57ace3e65b0be P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:49.160064 3000 leader_election.cc:290] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:49.160192 3109 rpcz_store.cc:276] 0112 02:52:45.758952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:45.758995 (+ 43us) service_pool.cc:225] Handling call
0112 02:52:49.077977 (+3318982us) ts_tablet_manager.cc:1145] Deleting tablet 7b9c845571f44c8eb8753090d63df45f
0112 02:52:49.077985 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.121010 (+ 43025us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.151345 (+ 30335us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.154403 (+ 3058us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3318710}
I20250112 02:52:49.160522 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 430a8652c53e4801a1c72de529c2adcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:49.160710 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 489) took 3151 ms. Trace:
I20250112 02:52:49.160871 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 475b8ea7d68b44fba7431a2812c13871 with delete_type TABLET_DATA_DELETED (replaced by tablet 0048bafbc21e4905864aa40f5daa2d7b) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:49.160815 3109 rpcz_store.cc:276] 0112 02:52:46.009597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.009889 (+ 292us) service_pool.cc:225] Handling call
0112 02:52:49.160562 (+3150673us) ts_tablet_manager.cc:1145] Deleting tablet f60626faf203441ab48c60147ddc3cf8
0112 02:52:49.160571 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.160697 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3129395}
I20250112 02:52:49.161312 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430a8652c53e4801a1c72de529c2adcb" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:49.161377 748 tablet_service.cc:1467] Processing CreateTablet for tablet c69e38fcc8bb4868a5c5830edcafad6a (DEFAULT_TABLE table=default.Table_d9871dbfe3f84ae88ade407167ea927e [id=220fbde75b4b425e84bb77efc68d1fbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.161422 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 491) took 2980 ms. Trace:
I20250112 02:52:49.161486 3109 rpcz_store.cc:276] 0112 02:52:46.181132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.183534 (+ 2402us) service_pool.cc:225] Handling call
0112 02:52:49.161308 (+2977774us) ts_tablet_manager.cc:1145] Deleting tablet 37d38f5e525440f8a0f7822620933435
0112 02:52:49.161320 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.161409 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2977606}
I20250112 02:52:49.161693 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69e38fcc8bb4868a5c5830edcafad6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.161753 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 492) took 2980 ms. Trace:
I20250112 02:52:49.162917 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 6ff7a7cc250c446db0985ffc1bcc2f91
I20250112 02:52:49.163057 3109 rpcz_store.cc:276] 0112 02:52:46.181238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.183741 (+ 2503us) service_pool.cc:225] Handling call
0112 02:52:49.161671 (+2977930us) ts_tablet_manager.cc:1145] Deleting tablet cd52261c4a414e7d93a528e2eddfa9db
0112 02:52:49.161679 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.161741 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2977801}
I20250112 02:52:49.163455 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 490) took 2982 ms. Trace:
I20250112 02:52:49.163697 3109 rpcz_store.cc:276] 0112 02:52:46.181023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.187350 (+ 6327us) service_pool.cc:225] Handling call
0112 02:52:49.163359 (+2976009us) ts_tablet_manager.cc:1145] Deleting tablet d2028226498b4b41a95c7c836fe03b4d
0112 02:52:49.163371 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.163442 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2975706}
I20250112 02:52:49.163772 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet a243d008c17a46668a3d40e015e84f11
I20250112 02:52:49.166059 3109 tablet_replica.cc:331] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:49.166272 3109 raft_consensus.cc:2238] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:49.113682 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d47f6d663934b25a0200ee9fa81165d" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:49.114396 3348 tablet.cc:1721] T b9d525dae23744a4adfc2f58cb9945ce P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:49.167760 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_340460709c7c4e2f8beeb40dc774e80f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:49.168171 3109 pending_rounds.cc:62] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:49.114552 3332 tablet.cc:1721] T afd6d31155034908ac44a90c3a2ea7d6 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 02:52:49.168229 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_340460709c7c4e2f8beeb40dc774e80f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.122717 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 54ad5fa647744674832024a5decc6290 (DEFAULT_TABLE table=default.Table_a9249382d49d479bb4af7de4d2273661 [id=e6c67a7945ca4440b4b294f6b0c8c3fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.168588 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54ad5fa647744674832024a5decc6290. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.169389 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_df79501b6f344b14895bf5e731fcf1bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:49.169770 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df79501b6f344b14895bf5e731fcf1bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.170823 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_98f7915b6ff944c5859b87b36cb3bfd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:49.171213 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98f7915b6ff944c5859b87b36cb3bfd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.123062 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 498) took 2200 ms. Trace:
I20250112 02:52:49.171753 3111 rpcz_store.cc:276] 0112 02:52:46.922644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.925341 (+ 2697us) service_pool.cc:225] Handling call
0112 02:52:49.110190 (+2184849us) ts_tablet_manager.cc:1145] Deleting tablet 648960d4f08f4ce8a4f00aeeb34048f5
0112 02:52:49.122964 (+ 12774us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.123048 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2184484}
I20250112 02:52:49.123296 1173 tablet_bootstrap.cc:492] T 5450777104da48f1821575658fd31afd P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:49.172293 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_e859bd6a7ded4d74b9eaa508d70c27e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:49.172657 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e859bd6a7ded4d74b9eaa508d70c27e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.173189 1173 tablet_bootstrap.cc:654] T 5450777104da48f1821575658fd31afd P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.173655 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 6151006e155440b7b02ab5e927805101
I20250112 02:52:49.123633 989 leader_election.cc:304] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
W20250112 02:52:49.173929 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 50311.1us
I20250112 02:52:49.174351 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_72fe7f15d46f4d03b77b3aa286d74c78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:49.174700 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72fe7f15d46f4d03b77b3aa286d74c78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.174808 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet a243d008c17a46668a3d40e015e84f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:49.175101 3347 raft_consensus.cc:2798] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:49.175195 3347 raft_consensus.cc:695] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:49.175400 3347 consensus_queue.cc:237] T 2ecf730b6985404a8134b497a4b792c8 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.176383 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ff7a7cc250c446db0985ffc1bcc2f91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:49.176548 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 6151006e155440b7b02ab5e927805101 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:49.177349 888 tablet_service.cc:1514] Processing DeleteTablet for tablet a243d008c17a46668a3d40e015e84f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:38786
W20250112 02:52:49.177433 855 leader_election.cc:343] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.177549 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 6151006e155440b7b02ab5e927805101 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:49.177978 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ff7a7cc250c446db0985ffc1bcc2f91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:49.177999 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_630d49265f684b0dbef983cb09ea0a14" } new_table_name: "default.Table_46e620b9d6e94d55838f20d265fc16cb" modify_external_catalogs: true
W20250112 02:52:49.178123 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 648960d4f08f4ce8a4f00aeeb34048f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 648960d4f08f4ce8a4f00aeeb34048f5 already in progress: creating tablet
I20250112 02:52:49.178429 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 501) took 2250 ms. Trace:
I20250112 02:52:49.178505 3111 rpcz_store.cc:276] 0112 02:52:46.928178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.928476 (+ 298us) service_pool.cc:225] Handling call
0112 02:52:49.171967 (+2243491us) ts_tablet_manager.cc:1145] Deleting tablet 2796289c7e8f4b45b26508c50b57c640
0112 02:52:49.178310 (+ 6343us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.178414 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2240883}
I20250112 02:52:49.178741 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430a8652c53e4801a1c72de529c2adcb" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:49.180697 3111 tablet_replica.cc:331] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:49.180894 3111 raft_consensus.cc:2238] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:49.181145 3111 raft_consensus.cc:2267] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:49.181608 3111 ts_tablet_manager.cc:1905] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:49.182667 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 9bbef42e11a843b18ed582ab4d04a92a
I20250112 02:52:49.183176 455 catalog_manager.cc:6596] ReplaceTablet: tablet 9bbef42e11a843b18ed582ab4d04a92a deleted and replaced by tablet 55acc4d7f7144f74a298e590f4932fc4
I20250112 02:52:49.183568 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_5ee93273ebaf4dfca589165e957c53d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:49.184087 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ee93273ebaf4dfca589165e957c53d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.184733 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bbef42e11a843b18ed582ab4d04a92a with delete_type TABLET_DATA_DELETED (replaced by tablet 55acc4d7f7144f74a298e590f4932fc4) from {username='slave'} at 127.0.0.1:56348
W20250112 02:52:49.185196 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 2796289c7e8f4b45b26508c50b57c640 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2796289c7e8f4b45b26508c50b57c640 already in progress: creating tablet
W20250112 02:52:49.185457 854 leader_election.cc:343] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.185551 854 leader_election.cc:304] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:49.186187 3000 raft_consensus.cc:2743] T 430a8652c53e4801a1c72de529c2adcb P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:49.187377 452 catalog_manager.cc:5526] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:49.188347 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 2a6650515ea540a8bbf21e2f88363360 (table default.Table_eaf6d6a8cdc84bed8248e8af835d85b0 [id=aed93c71d8e74a4990025965344aaab4]). Expected version 1 got 0
I20250112 02:52:49.188680 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet cec5346e13164274ae7bc3b6aa241f61 (table default.Table_04503e959a3c418a92dbb16260fbdc5c [id=8dd987fcd63c4ee7a5fcabd32affaf3d]). Expected version 1 got 0
I20250112 02:52:49.188989 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 9b81aee21510452881948e41d7aba9be (table default.Table_facdca78b120425892df7fd49aad6ea6 [id=8b6988397e544a37b869f1e782095b64]). Expected version 1 got 0
I20250112 02:52:49.189229 452 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:49.190292 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bbef42e11a843b18ed582ab4d04a92a with delete_type TABLET_DATA_DELETED (replaced by tablet 55acc4d7f7144f74a298e590f4932fc4) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:49.190886 3111 ts_tablet_manager.cc:1918] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:49.190985 3111 log.cc:1198] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/777272e7159b438ebff4db93698411d0
I20250112 02:52:49.191329 3111 ts_tablet_manager.cc:1939] T 777272e7159b438ebff4db93698411d0 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:49.192030 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_321da1daec2f410788a2a9611dd3d40a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:49.192463 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_321da1daec2f410788a2a9611dd3d40a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.193220 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 22435e0a54c1491eba1d9965099a4f3a (DEFAULT_TABLE table=default.Table_4215db3e1bc34b188fdda22e81f7d325 [id=14cc1c1d1533428d808141941b70af42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.193527 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22435e0a54c1491eba1d9965099a4f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.193965 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_e1d4a3104fc0481287acede4261941c6" } new_table_name: "default.Table_05982680822443a6aab5be202c2d8204" modify_external_catalogs: true
I20250112 02:52:49.194250 895 tablet_service.cc:1467] Processing CreateTablet for tablet 54ad5fa647744674832024a5decc6290 (DEFAULT_TABLE table=default.Table_a9249382d49d479bb4af7de4d2273661 [id=e6c67a7945ca4440b4b294f6b0c8c3fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.194574 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54ad5fa647744674832024a5decc6290. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.195847 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_1325d9a6747941458d8655d565170565" } modify_external_catalogs: true
I20250112 02:52:49.196014 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_1325d9a6747941458d8655d565170565" } modify_external_catalogs: true
I20250112 02:52:49.197290 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_c3b58715b551498fbff40d4478fa3ad9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:49.197681 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3b58715b551498fbff40d4478fa3ad9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.199276 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_b32fcf8b32d44d2093696313ffcb2083" } new_table_name: "default.Table_6aa6eabc06d84ff2bc9d6f28dd4dafc9" modify_external_catalogs: true
I20250112 02:52:49.203583 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_d9117b8cf65240dab1d9c1f61fe05192 [id=56974bbf7ff24cf0b2e9cf834e1c99f0] alter complete (version 2)
I20250112 02:52:49.214349 886 tablet_service.cc:1467] Processing CreateTablet for tablet 22435e0a54c1491eba1d9965099a4f3a (DEFAULT_TABLE table=default.Table_4215db3e1bc34b188fdda22e81f7d325 [id=14cc1c1d1533428d808141941b70af42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.214708 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22435e0a54c1491eba1d9965099a4f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.180505 3282 raft_consensus.cc:357] T 896908b2726c40f9aabdfc9a6b3283f7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.220273 1181 tablet_bootstrap.cc:492] T 1164804f2e2644b589ff727f499ed6a1 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:49.220412 1181 ts_tablet_manager.cc:1397] T 1164804f2e2644b589ff727f499ed6a1 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.091s user 0.009s sys 0.000s
I20250112 02:52:49.221220 1181 raft_consensus.cc:357] T 1164804f2e2644b589ff727f499ed6a1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.221422 1181 raft_consensus.cc:383] T 1164804f2e2644b589ff727f499ed6a1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.221468 1181 raft_consensus.cc:738] T 1164804f2e2644b589ff727f499ed6a1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.221654 1181 consensus_queue.cc:260] T 1164804f2e2644b589ff727f499ed6a1 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.222174 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 856fa83dedfd4b049e3e0d91d7ac5e40
I20250112 02:52:49.222786 1181 ts_tablet_manager.cc:1428] T 1164804f2e2644b589ff727f499ed6a1 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:49.223318 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 856fa83dedfd4b049e3e0d91d7ac5e40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:49 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:49.223902 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 856fa83dedfd4b049e3e0d91d7ac5e40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:49 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:49.226891 1181 tablet_bootstrap.cc:492] T bd6272f9d0754f0cbc1fb78391f0f927 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:49.228250 1181 tablet_bootstrap.cc:654] T bd6272f9d0754f0cbc1fb78391f0f927 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.233768 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 50e92c7a81194f38ab56501c03c01c45 (DEFAULT_TABLE table=default.Table_45898d7699014943b4a0d8d270a219ef [id=3860aab3bd764687a104423062adc462]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.234170 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e92c7a81194f38ab56501c03c01c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.237794 3282 raft_consensus.cc:738] T 896908b2726c40f9aabdfc9a6b3283f7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.238133 3282 consensus_queue.cc:260] T 896908b2726c40f9aabdfc9a6b3283f7 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.238660 3282 ts_tablet_manager.cc:1428] T 896908b2726c40f9aabdfc9a6b3283f7 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.059s user 0.002s sys 0.000s
I20250112 02:52:49.242193 886 tablet_service.cc:1467] Processing CreateTablet for tablet 50e92c7a81194f38ab56501c03c01c45 (DEFAULT_TABLE table=default.Table_45898d7699014943b4a0d8d270a219ef [id=3860aab3bd764687a104423062adc462]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.242911 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e92c7a81194f38ab56501c03c01c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.243443 888 tablet_service.cc:1467] Processing CreateTablet for tablet a7e5f52371b24a4ead963a4ae1d81604 (DEFAULT_TABLE table=default.Table_bdcd9ae330ee40d7881d369688794d8b [id=e15dd6f674b3420c893779d65b499408]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.243809 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7e5f52371b24a4ead963a4ae1d81604. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.243804 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 7b9c845571f44c8eb8753090d63df45f (table default.Table_7077f9a718664d8b915ee4003502306a [id=6df53b1d0c7a4fa1a18b3202741630b5]) successfully deleted
W20250112 02:52:49.244217 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f60626faf203441ab48c60147ddc3cf8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f60626faf203441ab48c60147ddc3cf8 already in progress: creating tablet
W20250112 02:52:49.244547 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 37d38f5e525440f8a0f7822620933435 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37d38f5e525440f8a0f7822620933435 already in progress: creating tablet
W20250112 02:52:49.245159 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet d2028226498b4b41a95c7c836fe03b4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2028226498b4b41a95c7c836fe03b4d already in progress: creating tablet
I20250112 02:52:49.247337 455 master_service.cc:946] ReplaceTablet: received request to replace tablet a437952889494b0e820e41fdfffe708d from {username='slave'} at 127.0.0.1:40610
I20250112 02:52:49.249117 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 394) took 2732 ms. Trace:
I20250112 02:52:49.249214 3110 rpcz_store.cc:276] 0112 02:52:46.516166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.516299 (+ 133us) service_pool.cc:225] Handling call
0112 02:52:49.139513 (+2623214us) ts_tablet_manager.cc:1145] Deleting tablet d0cb54003a83472cb9c57ace3e65b0be
0112 02:52:49.139520 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.151706 (+ 12186us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.159633 (+ 7927us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.249100 (+ 89467us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2623128}
I20250112 02:52:49.249655 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet d0cb54003a83472cb9c57ace3e65b0be (table default.Table_cb4ed576320c497588e65dcbac70afe9 [id=eee3e8b1198e401c93b8016a7c69b525]) successfully deleted
W20250112 02:52:49.252210 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7deb483f2e7c4a1683185db518d171e8 consensus state has no leader
I20250112 02:52:49.256742 1181 tablet_bootstrap.cc:492] T bd6272f9d0754f0cbc1fb78391f0f927 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:49.256914 1181 ts_tablet_manager.cc:1397] T bd6272f9d0754f0cbc1fb78391f0f927 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250112 02:52:49.257577 1181 raft_consensus.cc:357] T bd6272f9d0754f0cbc1fb78391f0f927 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.257802 1181 raft_consensus.cc:383] T bd6272f9d0754f0cbc1fb78391f0f927 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.257907 1181 raft_consensus.cc:738] T bd6272f9d0754f0cbc1fb78391f0f927 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.258152 1181 consensus_queue.cc:260] T bd6272f9d0754f0cbc1fb78391f0f927 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.258713 1181 ts_tablet_manager.cc:1428] T bd6272f9d0754f0cbc1fb78391f0f927 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:49.258980 1181 tablet_bootstrap.cc:492] T 262b5fa574fd4baa83aed9d1cf08cdca P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:49.239955 3310 tablet_copy_client.cc:806] T 05000c6a48e0450d8e6685ca864f67f0 P f1e775296eb746db8b318f17d26820e6: tablet copy: Starting download of 0 data blocks...
I20250112 02:52:49.259773 3310 tablet_copy_client.cc:670] T 05000c6a48e0450d8e6685ca864f67f0 P f1e775296eb746db8b318f17d26820e6: tablet copy: Starting download of 1 WAL segments...
I20250112 02:52:49.246096 1049 raft_consensus.cc:1212] T b7340a7a916c4b2fab32e0ba2d0ce322 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20250112 02:52:49.247586 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d47f6d663934b25a0200ee9fa81165d" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:49.248054 1053 raft_consensus.cc:1212] T ca6f080e5fd84084b601b646d4e6c9e2 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.3] Dedup: 1.2->[1.3-1.3]
I20250112 02:52:49.260854 3332 tablet.cc:1721] T ca6f080e5fd84084b601b646d4e6c9e2 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:49.262221 3000 raft_consensus.cc:491] T bd6272f9d0754f0cbc1fb78391f0f927 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.262337 3000 raft_consensus.cc:513] T bd6272f9d0754f0cbc1fb78391f0f927 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.262815 3000 leader_election.cc:290] T bd6272f9d0754f0cbc1fb78391f0f927 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:49.263170 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6272f9d0754f0cbc1fb78391f0f927" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:49.263875 854 leader_election.cc:343] T bd6272f9d0754f0cbc1fb78391f0f927 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.264083 3364 consensus_queue.cc:1035] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:49.266660 1044 raft_consensus.cc:1212] T 5804fc4130f84c69832c1022b7bd7fa5 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20250112 02:52:49.269729 1030 tablet_service.cc:1467] Processing CreateTablet for tablet a07f3cc538794990a491cba2e798844f (DEFAULT_TABLE table=default.Table_4194f66a2f28431c89e268ef7df1bb66 [id=71e607c245634fb19b7a68ae6cbe4c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.270046 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a07f3cc538794990a491cba2e798844f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.274251 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_67b6f55a9ba84491bdda41cdf82c25fc [id=63c20c66463d4a30b7e1558fecce2704] alter complete (version 4)
I20250112 02:52:49.251305 1019 tablet_service.cc:1467] Processing CreateTablet for tablet a7e5f52371b24a4ead963a4ae1d81604 (DEFAULT_TABLE table=default.Table_bdcd9ae330ee40d7881d369688794d8b [id=e15dd6f674b3420c893779d65b499408]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.280485 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7e5f52371b24a4ead963a4ae1d81604. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:49.281893 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 57813.9us
I20250112 02:52:49.284428 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 393) took 2768 ms. Trace:
I20250112 02:52:49.286396 3110 rpcz_store.cc:276] 0112 02:52:46.516075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.516222 (+ 147us) service_pool.cc:225] Handling call
0112 02:52:49.284319 (+2768097us) ts_tablet_manager.cc:1145] Deleting tablet 93aba9b4d1dd485cb62c4827c916a1d5
0112 02:52:49.284332 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.284406 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2766524}
I20250112 02:52:49.288324 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_f8ab3098ff7f4c479588888eac14e9a7" } modify_external_catalogs: true
I20250112 02:52:49.288411 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 6a49cd3f35b541c4bc9d5dd47a9416b9 (DEFAULT_TABLE table=default.Table_65c1669814384f6bb9da3a3b1e982d33 [id=f455355a632e421e90d87cc552d18ffc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.288534 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_f8ab3098ff7f4c479588888eac14e9a7" } modify_external_catalogs: true
I20250112 02:52:49.288733 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a49cd3f35b541c4bc9d5dd47a9416b9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:49.290818 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 93aba9b4d1dd485cb62c4827c916a1d5 because the tablet was not found. No further retry: Not found: Tablet not found: 93aba9b4d1dd485cb62c4827c916a1d5
I20250112 02:52:49.291735 3000 raft_consensus.cc:491] T f60626faf203441ab48c60147ddc3cf8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.291852 3000 raft_consensus.cc:513] T f60626faf203441ab48c60147ddc3cf8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.292408 3000 leader_election.cc:290] T f60626faf203441ab48c60147ddc3cf8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:49.292747 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f60626faf203441ab48c60147ddc3cf8" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:49.293184 854 leader_election.cc:343] T f60626faf203441ab48c60147ddc3cf8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.293478 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 396) took 2775 ms. Trace:
I20250112 02:52:49.293558 3110 rpcz_store.cc:276] 0112 02:52:46.518031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.518092 (+ 61us) service_pool.cc:225] Handling call
0112 02:52:49.293402 (+2775310us) ts_tablet_manager.cc:1145] Deleting tablet 940affdc2c484b7d9fd2de17f9d1cea7
0112 02:52:49.293411 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.293464 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2775180}
W20250112 02:52:49.294004 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet 940affdc2c484b7d9fd2de17f9d1cea7 because the tablet was not found. No further retry: Not found: Tablet not found: 940affdc2c484b7d9fd2de17f9d1cea7
I20250112 02:52:49.294476 895 tablet_service.cc:1467] Processing CreateTablet for tablet a07f3cc538794990a491cba2e798844f (DEFAULT_TABLE table=default.Table_4194f66a2f28431c89e268ef7df1bb66 [id=71e607c245634fb19b7a68ae6cbe4c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.300092 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a07f3cc538794990a491cba2e798844f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.301698 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 0901eb8b296d40eba419b2a1700014f9 (DEFAULT_TABLE table=default.Table_9d1a5f1a606f4ea886dcf4616f6036d5 [id=1ff7d0e5b2b44e9eb8f04eda9e3d3068]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.302004 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0901eb8b296d40eba419b2a1700014f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.302454 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_9035b1f0824d486aa993f6bc318ed623 [id=3a0f5ac3329d4b60befce41c91f5ae5c] alter complete (version 2)
I20250112 02:52:49.300117 888 tablet_service.cc:1467] Processing CreateTablet for tablet 6a49cd3f35b541c4bc9d5dd47a9416b9 (DEFAULT_TABLE table=default.Table_65c1669814384f6bb9da3a3b1e982d33 [id=f455355a632e421e90d87cc552d18ffc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.301555 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 395) took 2783 ms. Trace:
I20250112 02:52:49.303727 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a49cd3f35b541c4bc9d5dd47a9416b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.254434 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 510) took 2243 ms. Trace:
I20250112 02:52:49.240278 3266 raft_consensus.cc:491] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.305493 3266 raft_consensus.cc:513] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.306027 3266 leader_election.cc:290] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:49.307029 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet a437952889494b0e820e41fdfffe708d
I20250112 02:52:49.307416 455 catalog_manager.cc:6596] ReplaceTablet: tablet a437952889494b0e820e41fdfffe708d deleted and replaced by tablet b8a8c5a132674057a188b73dcdea0dfd
I20250112 02:52:49.306980 3111 rpcz_store.cc:276] 0112 02:52:47.010536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.010585 (+ 49us) service_pool.cc:225] Handling call
0112 02:52:49.178704 (+2168119us) ts_tablet_manager.cc:1145] Deleting tablet 777272e7159b438ebff4db93698411d0
0112 02:52:49.180635 (+ 1931us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.188159 (+ 7524us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.190813 (+ 2654us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.254418 (+ 63605us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2167883}
I20250112 02:52:49.307565 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet a437952889494b0e820e41fdfffe708d with delete_type TABLET_DATA_DELETED (replaced by tablet b8a8c5a132674057a188b73dcdea0dfd) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:49.307590 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eed0a9f8354e5391ec2463dafdb683" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:49.307752 909 raft_consensus.cc:2463] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:49.308487 989 leader_election.cc:304] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:49.309095 3266 raft_consensus.cc:2798] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:49.309160 3266 raft_consensus.cc:491] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:49.309211 3266 raft_consensus.cc:3054] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:49.303130 3110 rpcz_store.cc:276] 0112 02:52:46.517938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.518174 (+ 236us) service_pool.cc:225] Handling call
0112 02:52:49.293817 (+2775643us) ts_tablet_manager.cc:1145] Deleting tablet f3771166b77e4f90aaa116ab1a9dc299
0112 02:52:49.293824 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.301539 (+ 7715us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2775509}
W20250112 02:52:49.315515 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet f3771166b77e4f90aaa116ab1a9dc299 because the tablet was not found. No further retry: Not found: Tablet not found: f3771166b77e4f90aaa116ab1a9dc299
I20250112 02:52:49.316282 888 tablet_service.cc:1514] Processing DeleteTablet for tablet a437952889494b0e820e41fdfffe708d with delete_type TABLET_DATA_DELETED (replaced by tablet b8a8c5a132674057a188b73dcdea0dfd) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:49.316509 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 397) took 2797 ms. Trace:
I20250112 02:52:49.317296 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0a7cada26c450abbb65a1ec932324e (DEFAULT_TABLE table=default.Table_bd3acd3ed2f141db9e0640f76305391b [id=cfee1111082d49478d3e391da08d5584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.317668 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a0a7cada26c450abbb65a1ec932324e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.317318 3110 rpcz_store.cc:276] 0112 02:52:46.518135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.518286 (+ 151us) service_pool.cc:225] Handling call
0112 02:52:49.315508 (+2797222us) ts_tablet_manager.cc:1145] Deleting tablet a8e7882e07784bdda18dc1ffd20963c8
0112 02:52:49.315517 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.315584 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2797087}
W20250112 02:52:49.320691 440 catalog_manager.cc:4839] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet a8e7882e07784bdda18dc1ffd20963c8 because the tablet was not found. No further retry: Not found: Tablet not found: a8e7882e07784bdda18dc1ffd20963c8
I20250112 02:52:49.185175 1177 tablet_bootstrap.cc:492] T f3771166b77e4f90aaa116ab1a9dc299 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:49.201684 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ff7a7cc250c446db0985ffc1bcc2f91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:49.321005 1177 ts_tablet_manager.cc:1397] T f3771166b77e4f90aaa116ab1a9dc299 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.201s user 0.003s sys 0.003s
I20250112 02:52:49.321631 1177 raft_consensus.cc:357] T f3771166b77e4f90aaa116ab1a9dc299 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.321790 1177 raft_consensus.cc:383] T f3771166b77e4f90aaa116ab1a9dc299 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.321830 1177 raft_consensus.cc:738] T f3771166b77e4f90aaa116ab1a9dc299 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.321978 1177 consensus_queue.cc:260] T f3771166b77e4f90aaa116ab1a9dc299 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.322129 888 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0a7cada26c450abbb65a1ec932324e (DEFAULT_TABLE table=default.Table_bd3acd3ed2f141db9e0640f76305391b [id=cfee1111082d49478d3e391da08d5584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.322384 1177 ts_tablet_manager.cc:1428] T f3771166b77e4f90aaa116ab1a9dc299 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 02:52:49.322517 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_ac4a10685c414bef86c147c336335ff7" } new_table_name: "default.Table_78c27e8312b94a05bfe54db6e0aeb659" modify_external_catalogs: true
I20250112 02:52:49.322607 1177 tablet_bootstrap.cc:492] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:49.201891 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bbef42e11a843b18ed582ab4d04a92a with delete_type TABLET_DATA_DELETED (replaced by tablet 55acc4d7f7144f74a298e590f4932fc4) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:49.202446 752 tablet_service.cc:1514] Processing DeleteTablet for tablet a243d008c17a46668a3d40e015e84f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:49.202562 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 6151006e155440b7b02ab5e927805101 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:48 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:49.324780 1177 tablet_bootstrap.cc:654] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.204329 743 tablet_service.cc:1467] Processing CreateTablet for tablet 54ad5fa647744674832024a5decc6290 (DEFAULT_TABLE table=default.Table_a9249382d49d479bb4af7de4d2273661 [id=e6c67a7945ca4440b4b294f6b0c8c3fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.325899 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54ad5fa647744674832024a5decc6290. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.329731 889 tablet_service.cc:1467] Processing CreateTablet for tablet acdc70e0fe394ff2a78a3e0cb8587871 (DEFAULT_TABLE table=default.Table_9a6eb566a9f24077bb60c1c5434165e4 [id=bcdf830a7feb475abb76e6ebd1467f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.331139 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_ef71f75b40b14fa5b96ee12db588b5d8 [id=8cc378b0da6e4b77bc2c6822cd0844fe] alter complete (version 1)
I20250112 02:52:49.331699 1016 tablet_service.cc:1467] Processing CreateTablet for tablet acdc70e0fe394ff2a78a3e0cb8587871 (DEFAULT_TABLE table=default.Table_9a6eb566a9f24077bb60c1c5434165e4 [id=bcdf830a7feb475abb76e6ebd1467f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.332012 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acdc70e0fe394ff2a78a3e0cb8587871. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.220093 748 tablet_service.cc:1467] Processing CreateTablet for tablet 22435e0a54c1491eba1d9965099a4f3a (DEFAULT_TABLE table=default.Table_4215db3e1bc34b188fdda22e81f7d325 [id=14cc1c1d1533428d808141941b70af42]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:49.232950 713 consensus_peers.cc:487] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c29a9ca43e034b43a00207e5847bbee3. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:49.258601 799 tablet_copy_service.cc:342] P 289afc116aa74408aa5bff74e11c50d8: Request end of tablet copy session f1e775296eb746db8b318f17d26820e6-896908b2726c40f9aabdfc9a6b3283f7 received from {username='slave'} at 127.0.104.3:52505
W20250112 02:52:49.334805 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:44451, user_credentials={real_user=slave}} blocked reactor thread for 101872us
I20250112 02:52:49.334879 799 tablet_copy_service.cc:434] P 289afc116aa74408aa5bff74e11c50d8: ending tablet copy session f1e775296eb746db8b318f17d26820e6-896908b2726c40f9aabdfc9a6b3283f7 on tablet 896908b2726c40f9aabdfc9a6b3283f7 with peer f1e775296eb746db8b318f17d26820e6
I20250112 02:52:49.335008 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 51155a89d1b6442e990d8a3de7756a87 (DEFAULT_TABLE table=default.Table_3140af67d0f7472ea77365c505900416 [id=a0d79330a98447029ef10c8d550f64c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.335283 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51155a89d1b6442e990d8a3de7756a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.263870 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6272f9d0754f0cbc1fb78391f0f927" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:49.336143 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22435e0a54c1491eba1d9965099a4f3a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:49.340390 855 leader_election.cc:343] T bd6272f9d0754f0cbc1fb78391f0f927 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.340520 855 leader_election.cc:304] T bd6272f9d0754f0cbc1fb78391f0f927 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:49.340696 3273 raft_consensus.cc:491] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.340775 3273 raft_consensus.cc:513] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.341229 3273 leader_election.cc:290] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:49.341328 3273 raft_consensus.cc:2743] T bd6272f9d0754f0cbc1fb78391f0f927 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:49.342043 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c12a1c5e3584450ac193743a0107e80" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:49.342504 855 leader_election.cc:343] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.343461 3310 tablet_copy_client.cc:538] T 05000c6a48e0450d8e6685ca864f67f0 P f1e775296eb746db8b318f17d26820e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 02:52:49.346851 3310 tablet_bootstrap.cc:492] T 05000c6a48e0450d8e6685ca864f67f0 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:49.287415 714 leader_election.cc:343] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.292939 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f60626faf203441ab48c60147ddc3cf8" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:49.306571 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eed0a9f8354e5391ec2463dafdb683" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:49.347826 714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 60412.3us
I20250112 02:52:49.219877 760 tablet_service.cc:1467] Processing CreateTablet for tablet 50e92c7a81194f38ab56501c03c01c45 (DEFAULT_TABLE table=default.Table_45898d7699014943b4a0d8d270a219ef [id=3860aab3bd764687a104423062adc462]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:49.348050 714 consensus_peers.cc:487] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c29a9ca43e034b43a00207e5847bbee3. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:49.348327 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e92c7a81194f38ab56501c03c01c45. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:52:49.348457 714 leader_election.cc:343] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.350437 909 raft_consensus.cc:1270] T 0f16ec377f01484eae10e63490da143f P fe36951a83fb4e57b26e187edbcbfe3d [term 2 FOLLOWER]: Refusing update from remote peer 289afc116aa74408aa5bff74e11c50d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250112 02:52:49.350839 3106 consensus_queue.cc:1035] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:52:49.351636 855 leader_election.cc:343] T f60626faf203441ab48c60147ddc3cf8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.351779 855 leader_election.cc:304] T f60626faf203441ab48c60147ddc3cf8 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
W20250112 02:52:49.351779 991 leader_election.cc:343] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.352176 3000 raft_consensus.cc:2743] T f60626faf203441ab48c60147ddc3cf8 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:49.357806 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c12a1c5e3584450ac193743a0107e80" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:49.360541 854 leader_election.cc:343] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.360678 854 leader_election.cc:304] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:49.360910 3000 raft_consensus.cc:2743] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:49.366104 3307 ts_tablet_manager.cc:927] T 24ebf85f76f44633a20d97964638c0ac P f1e775296eb746db8b318f17d26820e6: Initiating tablet copy from peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007)
I20250112 02:52:49.367393 3109 pending_rounds.cc:69] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319911478276096 op_type: NO_OP noop_request { }
I20250112 02:52:49.367568 3109 raft_consensus.cc:2883] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:49.368065 3109 raft_consensus.cc:2267] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:49.370018 3109 ts_tablet_manager.cc:1905] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:49.373024 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 513) took 2298 ms. Trace:
I20250112 02:52:49.373112 3111 rpcz_store.cc:276] 0112 02:52:47.074601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.075906 (+ 1305us) service_pool.cc:225] Handling call
0112 02:52:49.372919 (+2297013us) ts_tablet_manager.cc:1145] Deleting tablet 9fae49da375f4c6a894c1237dd6621fb
0112 02:52:49.372927 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.373009 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2129309}
W20250112 02:52:49.373649 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 9fae49da375f4c6a894c1237dd6621fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fae49da375f4c6a894c1237dd6621fb already in progress: creating tablet
I20250112 02:52:49.373977 3111 tablet_replica.cc:331] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:49.374212 3111 raft_consensus.cc:2238] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:49.374890 3111 raft_consensus.cc:2267] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:49.377012 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 856fa83dedfd4b049e3e0d91d7ac5e40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:49 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:49.377547 747 tablet_service.cc:1467] Processing CreateTablet for tablet a7e5f52371b24a4ead963a4ae1d81604 (DEFAULT_TABLE table=default.Table_bdcd9ae330ee40d7881d369688794d8b [id=e15dd6f674b3420c893779d65b499408]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.377835 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7e5f52371b24a4ead963a4ae1d81604. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.385682 747 tablet_service.cc:1467] Processing CreateTablet for tablet a07f3cc538794990a491cba2e798844f (DEFAULT_TABLE table=default.Table_4194f66a2f28431c89e268ef7df1bb66 [id=71e607c245634fb19b7a68ae6cbe4c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.386034 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a07f3cc538794990a491cba2e798844f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.385970 3307 tablet_copy_client.cc:323] T 24ebf85f76f44633a20d97964638c0ac P f1e775296eb746db8b318f17d26820e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.104.1:35007
I20250112 02:52:49.386646 800 tablet_copy_service.cc:140] P 289afc116aa74408aa5bff74e11c50d8: Received BeginTabletCopySession request for tablet 24ebf85f76f44633a20d97964638c0ac from peer f1e775296eb746db8b318f17d26820e6 ({username='slave'} at 127.0.104.3:52505)
I20250112 02:52:49.386567 3266 raft_consensus.cc:513] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.386756 800 tablet_copy_service.cc:161] P 289afc116aa74408aa5bff74e11c50d8: Beginning new tablet copy session on tablet 24ebf85f76f44633a20d97964638c0ac from peer f1e775296eb746db8b318f17d26820e6 at {username='slave'} at 127.0.104.3:52505: session id = f1e775296eb746db8b318f17d26820e6-24ebf85f76f44633a20d97964638c0ac
I20250112 02:52:49.387104 3266 leader_election.cc:290] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:49.388720 3133 raft_consensus.cc:491] T f3771166b77e4f90aaa116ab1a9dc299 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.388820 3133 raft_consensus.cc:513] T f3771166b77e4f90aaa116ab1a9dc299 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.389180 800 tablet_copy_source_session.cc:215] T 24ebf85f76f44633a20d97964638c0ac P 289afc116aa74408aa5bff74e11c50d8: Tablet Copy: opened 0 blocks and 1 log segments
I20250112 02:52:49.389317 3133 leader_election.cc:290] T f3771166b77e4f90aaa116ab1a9dc299 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:49.389365 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eed0a9f8354e5391ec2463dafdb683" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
W20250112 02:52:49.390373 991 leader_election.cc:343] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.390486 904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eed0a9f8354e5391ec2463dafdb683" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:49.390663 904 raft_consensus.cc:3054] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:49.393254 904 raft_consensus.cc:2463] T 42eed0a9f8354e5391ec2463dafdb683 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
W20250112 02:52:49.393618 713 leader_election.cc:343] T f3771166b77e4f90aaa116ab1a9dc299 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: f3771166b77e4f90aaa116ab1a9dc299
I20250112 02:52:49.393780 989 leader_election.cc:304] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:49.394747 3266 raft_consensus.cc:2798] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:49.394861 3266 raft_consensus.cc:695] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:49.395061 3266 consensus_queue.cc:237] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.396639 3000 raft_consensus.cc:491] T 1164804f2e2644b589ff727f499ed6a1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.396760 3000 raft_consensus.cc:513] T 1164804f2e2644b589ff727f499ed6a1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.397253 3000 leader_election.cc:290] T 1164804f2e2644b589ff727f499ed6a1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:49.394335 3307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ebf85f76f44633a20d97964638c0ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.397814 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1164804f2e2644b589ff727f499ed6a1" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:49.398703 855 leader_election.cc:343] T 1164804f2e2644b589ff727f499ed6a1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.401772 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1164804f2e2644b589ff727f499ed6a1" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:49.402571 854 leader_election.cc:343] T 1164804f2e2644b589ff727f499ed6a1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.402721 854 leader_election.cc:304] T 1164804f2e2644b589ff727f499ed6a1 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:49.402973 3000 raft_consensus.cc:2743] T 1164804f2e2644b589ff727f499ed6a1 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:49.407132 3307 tablet_copy_client.cc:806] T 24ebf85f76f44633a20d97964638c0ac P f1e775296eb746db8b318f17d26820e6: tablet copy: Starting download of 0 data blocks...
I20250112 02:52:49.414230 752 tablet_service.cc:1467] Processing CreateTablet for tablet 6a49cd3f35b541c4bc9d5dd47a9416b9 (DEFAULT_TABLE table=default.Table_65c1669814384f6bb9da3a3b1e982d33 [id=f455355a632e421e90d87cc552d18ffc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.414567 755 tablet_service.cc:1514] Processing DeleteTablet for tablet a437952889494b0e820e41fdfffe708d with delete_type TABLET_DATA_DELETED (replaced by tablet b8a8c5a132674057a188b73dcdea0dfd) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:49.414701 745 tablet_service.cc:1467] Processing CreateTablet for tablet 0901eb8b296d40eba419b2a1700014f9 (DEFAULT_TABLE table=default.Table_9d1a5f1a606f4ea886dcf4616f6036d5 [id=1ff7d0e5b2b44e9eb8f04eda9e3d3068]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.414626 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a49cd3f35b541c4bc9d5dd47a9416b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.415004 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0901eb8b296d40eba419b2a1700014f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.419739 757 tablet_service.cc:1467] Processing CreateTablet for tablet acdc70e0fe394ff2a78a3e0cb8587871 (DEFAULT_TABLE table=default.Table_9a6eb566a9f24077bb60c1c5434165e4 [id=bcdf830a7feb475abb76e6ebd1467f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.420001 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acdc70e0fe394ff2a78a3e0cb8587871. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.421065 3110 tablet_replica.cc:331] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:49.421250 3110 raft_consensus.cc:2238] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:49.421372 3110 raft_consensus.cc:2267] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:49.421818 3110 ts_tablet_manager.cc:1905] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:49.423694 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acdc70e0fe394ff2a78a3e0cb8587871. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.423756 753 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0a7cada26c450abbb65a1ec932324e (DEFAULT_TABLE table=default.Table_bd3acd3ed2f141db9e0640f76305391b [id=cfee1111082d49478d3e391da08d5584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.424064 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a0a7cada26c450abbb65a1ec932324e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.430738 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a0a7cada26c450abbb65a1ec932324e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.435736 889 tablet_service.cc:1467] Processing CreateTablet for tablet 51155a89d1b6442e990d8a3de7756a87 (DEFAULT_TABLE table=default.Table_3140af67d0f7472ea77365c505900416 [id=a0d79330a98447029ef10c8d550f64c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.436064 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51155a89d1b6442e990d8a3de7756a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.438180 3374 tablet.cc:1721] T 223eb8b4d1024985844867d5641600d1 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:49.439152 3355 tablet.cc:1721] T 223eb8b4d1024985844867d5641600d1 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:49.447199 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_3406d849cc9a4ffab1b431da38b95bcf" } new_table_name: "default.Table_8fa574910a034a618c4ee6496a99fb9f" modify_external_catalogs: true
I20250112 02:52:49.448357 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet f5492416a0544951bcbf8a4f84942e66
I20250112 02:52:49.448503 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_7ceed202a8d94c0198eb7acdccf33a1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:49.449029 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ceed202a8d94c0198eb7acdccf33a1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.449138 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_2b29491143994d53a93a850ace3a9d7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:49.449551 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b29491143994d53a93a850ace3a9d7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.450114 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_e373fd8fe38d400f9bc33937fe89628b [id=ec37a56b10ba407b86e851f07bddf528] alter complete (version 1)
I20250112 02:52:49.451254 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_b58358f96eb64f048b6b222b9201c55f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:49.451694 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_1950cb8738a5407a84000b99c2cf5e2e" } modify_external_catalogs: true
I20250112 02:52:49.451869 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_1950cb8738a5407a84000b99c2cf5e2e" } modify_external_catalogs: true
I20250112 02:52:49.452939 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_ade612ff606d477c84a3de161a684cee" } modify_external_catalogs: true
I20250112 02:52:49.453078 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_ade612ff606d477c84a3de161a684cee" } modify_external_catalogs: true
I20250112 02:52:49.454074 893 tablet_service.cc:1514] Processing DeleteTablet for tablet f5492416a0544951bcbf8a4f84942e66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:49 UTC) from {username='slave'} at 127.0.0.1:38786
W20250112 02:52:49.455211 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b58358f96eb64f048b6b222b9201c55f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.457420 893 tablet_service.cc:1467] Processing CreateTablet for tablet d882205901e946a6b624c7aefc6a3921 (DEFAULT_TABLE table=default.Table_211137fe64404ad5ba736c0aea830162 [id=9545de1e92df466da0b695427cb996fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.457734 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d882205901e946a6b624c7aefc6a3921. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.465454 1181 tablet_bootstrap.cc:654] T 262b5fa574fd4baa83aed9d1cf08cdca P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.465852 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_9ea62863b8424bdfaef63e18436d9d51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:49.466264 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ea62863b8424bdfaef63e18436d9d51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.467204 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_8293882503214cf284e12ff8cc75b9c7" } modify_external_catalogs: true
I20250112 02:52:49.467363 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_8293882503214cf284e12ff8cc75b9c7" } modify_external_catalogs: true
I20250112 02:52:49.472208 893 tablet_service.cc:1467] Processing CreateTablet for tablet e1ac5b5ea3a6494baf806bd4441e747f (DEFAULT_TABLE table=default.Table_3f00edd22a8048609d508e9a9545a361 [id=b58c92930dda4f2c882eb730024971bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.472545 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1ac5b5ea3a6494baf806bd4441e747f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.490396 3355 tablet.cc:1721] T 2c03b5e612d64314b210c11686fdfa2a P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 8
I20250112 02:52:49.490841 3374 tablet.cc:1721] T 2c03b5e612d64314b210c11686fdfa2a P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 8
I20250112 02:52:49.494875 1177 tablet_bootstrap.cc:492] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:49.495060 1177 ts_tablet_manager.cc:1397] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.173s user 0.001s sys 0.009s
I20250112 02:52:49.507735 770 raft_consensus.cc:3054] T 3d418cda9ed5449fa45690130cd6d5bc P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:49.512867 885 tablet_service.cc:1467] Processing CreateTablet for tablet 083fcf0c7f584494832a3b05b0842dc3 (DEFAULT_TABLE table=default.Table_f5445f2c41c740a5bf3ceb49b8491359 [id=f9b9ddb521fa442eb7d60b47ed99679d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.513226 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 083fcf0c7f584494832a3b05b0842dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.513876 896 tablet_service.cc:1467] Processing CreateTablet for tablet 6506fb4f1dee4bd09f0f3cf52c6c1b2e (DEFAULT_TABLE table=default.Table_6b747ebd44244031a16f8b805caa9209 [id=6d83348798094235be56595652dccd6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.514187 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6506fb4f1dee4bd09f0f3cf52c6c1b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.519667 3000 raft_consensus.cc:491] T be856e4b913e40d58a3f3ba54698cc5f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.519780 3000 raft_consensus.cc:513] T be856e4b913e40d58a3f3ba54698cc5f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.520341 3000 leader_election.cc:290] T be856e4b913e40d58a3f3ba54698cc5f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:49.521023 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be856e4b913e40d58a3f3ba54698cc5f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:49.521600 855 leader_election.cc:343] T be856e4b913e40d58a3f3ba54698cc5f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.526103 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_f957cb19a384456eb663c27c470cf702 [id=3a629cb4a4104e30b0beb8f729ba0a22] alter complete (version 2)
W20250112 02:52:49.527205 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fc67986a45ed414296e68efac8e48fbf consensus state has no leader
I20250112 02:52:49.527894 3355 tablet.cc:1721] T 4916ae3917084cfe8c93240a8857be0e P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:49.532802 1181 tablet_bootstrap.cc:492] T 262b5fa574fd4baa83aed9d1cf08cdca P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:49.532932 1181 ts_tablet_manager.cc:1397] T 262b5fa574fd4baa83aed9d1cf08cdca P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.274s user 0.002s sys 0.000s
I20250112 02:52:49.533723 1181 raft_consensus.cc:357] T 262b5fa574fd4baa83aed9d1cf08cdca P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.533944 1181 raft_consensus.cc:383] T 262b5fa574fd4baa83aed9d1cf08cdca P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.534003 1181 raft_consensus.cc:738] T 262b5fa574fd4baa83aed9d1cf08cdca P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.534162 3374 tablet.cc:1721] T 4916ae3917084cfe8c93240a8857be0e P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:49.450325 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet f5492416a0544951bcbf8a4f84942e66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:49 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:49.457203 1022 tablet_service.cc:1467] Processing CreateTablet for tablet d882205901e946a6b624c7aefc6a3921 (DEFAULT_TABLE table=default.Table_211137fe64404ad5ba736c0aea830162 [id=9545de1e92df466da0b695427cb996fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.535338 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d882205901e946a6b624c7aefc6a3921. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.535933 448 catalog_manager.cc:5526] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 1 to 2, leader changed from f1e775296eb746db8b318f17d26820e6 (127.0.104.3) to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 2 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:49.536309 1181 consensus_queue.cc:260] T 262b5fa574fd4baa83aed9d1cf08cdca P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.536793 1181 ts_tablet_manager.cc:1428] T 262b5fa574fd4baa83aed9d1cf08cdca P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 02:52:49.537041 1181 tablet_bootstrap.cc:492] T cca2a096fce0464394f5029652813873 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:49.540567 1181 tablet_bootstrap.cc:654] T cca2a096fce0464394f5029652813873 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.541123 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet 0f16ec377f01484eae10e63490da143f (table default.Table_f20419242ba541528d0cf9ec37ac4e92 [id=63b58f300cd5448caa5c958d1299970e]). Expected version 1 got 0
I20250112 02:52:49.469123 3111 ts_tablet_manager.cc:1905] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:52:49.479964 3332 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.219s user 0.000s sys 0.000s
I20250112 02:52:49.542241 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet c5079f87652a4ea58b366d1f9448fd4e (table default.Table_c5704699453f40298d94c2f5dbd10179 [id=ed2c162c64f74c9f889789b3abe7c041]). Expected version 1 got 0
I20250112 02:52:49.542860 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet fc52271535df47e282bf1625c2e8e041 (table default.Table_efed5d5c2a7b421eba5e50bf60c00bce [id=e0767ae857dd4aac85e751c333ae51e0]). Expected version 1 got 0
I20250112 02:52:49.543378 448 catalog_manager.cc:5526] T 2d47f6d663934b25a0200ee9fa81165d P 289afc116aa74408aa5bff74e11c50d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "289afc116aa74408aa5bff74e11c50d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20250112 02:52:49.544051 448 catalog_manager.cc:5618] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:49.545329 3332 tablet.cc:1716] T ca6f080e5fd84084b601b646d4e6c9e2 P f1e775296eb746db8b318f17d26820e6: Skipping requested alter to schema version 3, tablet already version 3
W20250112 02:52:49.546859 714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 55224.1us
I20250112 02:52:49.547169 3307 tablet_copy_client.cc:670] T 24ebf85f76f44633a20d97964638c0ac P f1e775296eb746db8b318f17d26820e6: tablet copy: Starting download of 1 WAL segments...
I20250112 02:52:49.549542 1177 raft_consensus.cc:357] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.549708 1177 raft_consensus.cc:383] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.549753 1177 raft_consensus.cc:738] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.549917 1177 consensus_queue.cc:260] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.550011 3109 ts_tablet_manager.cc:1918] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:49.550103 3109 log.cc:1198] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/3499374810fb412e938325434c35618d
I20250112 02:52:49.550499 3109 ts_tablet_manager.cc:1939] T 3499374810fb412e938325434c35618d P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:49.550635 1177 ts_tablet_manager.cc:1428] T 3efcd32b4ccc4fada5adb3ca3bcdddaa P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.055s user 0.001s sys 0.000s
I20250112 02:52:49.550936 1177 tablet_bootstrap.cc:492] T 09900b42e8a443e680cc35c96a1dbbb7 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:49.552721 1177 tablet_bootstrap.cc:654] T 09900b42e8a443e680cc35c96a1dbbb7 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.556257 1181 tablet_bootstrap.cc:492] T cca2a096fce0464394f5029652813873 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:49.556375 1181 ts_tablet_manager.cc:1397] T cca2a096fce0464394f5029652813873 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.001s
I20250112 02:52:49.556977 1181 raft_consensus.cc:357] T cca2a096fce0464394f5029652813873 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.557154 1181 raft_consensus.cc:383] T cca2a096fce0464394f5029652813873 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.557222 1181 raft_consensus.cc:738] T cca2a096fce0464394f5029652813873 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.557561 1181 consensus_queue.cc:260] T cca2a096fce0464394f5029652813873 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.558199 1181 ts_tablet_manager.cc:1428] T cca2a096fce0464394f5029652813873 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:49.502032 1053 raft_consensus.cc:1270] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [term 2 FOLLOWER]: Refusing update from remote peer 289afc116aa74408aa5bff74e11c50d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250112 02:52:49.558427 1181 tablet_bootstrap.cc:492] T 6b1d46c5395d4defb2d0e2948807eaf4 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:49.558486 1053 pending_rounds.cc:77] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6: Aborting all ops after (but not including) 0
I20250112 02:52:49.558550 1053 pending_rounds.cc:99] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250112 02:52:49.558598 1053 raft_consensus.cc:2883] T 0f16ec377f01484eae10e63490da143f P f1e775296eb746db8b318f17d26820e6 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 02:52:49.559237 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_cf3097c8f56a45169250ebc2ecc6c35d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:49.559525 3307 tablet_copy_client.cc:538] T 24ebf85f76f44633a20d97964638c0ac P f1e775296eb746db8b318f17d26820e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250112 02:52:49.511101 3266 consensus_queue.cc:1035] T 3d418cda9ed5449fa45690130cd6d5bc P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:52:49.559746 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf3097c8f56a45169250ebc2ecc6c35d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.511304 3364 raft_consensus.cc:491] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.519520 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 6506fb4f1dee4bd09f0f3cf52c6c1b2e (DEFAULT_TABLE table=default.Table_6b747ebd44244031a16f8b805caa9209 [id=6d83348798094235be56595652dccd6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.560037 3364 raft_consensus.cc:513] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.560359 1181 tablet_bootstrap.cc:654] T 6b1d46c5395d4defb2d0e2948807eaf4 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.560629 3364 leader_election.cc:290] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:49.526886 3348 tablet.cc:1721] T 4916ae3917084cfe8c93240a8857be0e P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:49.561437 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61eebc5f22664712af6e74e82809ca87" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:49.561918 991 leader_election.cc:343] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.563588 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6506fb4f1dee4bd09f0f3cf52c6c1b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.567600 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61eebc5f22664712af6e74e82809ca87" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:49.567770 912 raft_consensus.cc:2463] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 0.
I20250112 02:52:49.567998 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_f89cd73745aa49dc955148070ed24e59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:49.568434 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f89cd73745aa49dc955148070ed24e59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.569787 989 leader_election.cc:304] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:49.570106 3266 raft_consensus.cc:2798] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:49.570179 3266 raft_consensus.cc:491] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:49.570230 3266 raft_consensus.cc:3054] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:49.571972 714 leader_election.cc:343] T f3771166b77e4f90aaa116ab1a9dc299 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: f3771166b77e4f90aaa116ab1a9dc299
I20250112 02:52:49.572135 714 leader_election.cc:304] T f3771166b77e4f90aaa116ab1a9dc299 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:49.572404 3106 raft_consensus.cc:2743] T f3771166b77e4f90aaa116ab1a9dc299 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:49.573501 3374 tablet.cc:1721] T e3ee5d8a17874d7ba54de51568e7d2e6 P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:49.574350 3106 consensus_queue.cc:1035] T 0f16ec377f01484eae10e63490da143f P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:49.576341 3266 raft_consensus.cc:513] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.577184 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61eebc5f22664712af6e74e82809ca87" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8"
W20250112 02:52:49.577633 991 leader_election.cc:343] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.577900 3266 leader_election.cc:290] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Requested vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451)
I20250112 02:52:49.580147 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61eebc5f22664712af6e74e82809ca87" candidate_uuid: "f1e775296eb746db8b318f17d26820e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:49.580303 912 raft_consensus.cc:3054] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:49.580708 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 67eca0da8346407fa4687738328dc414 (DEFAULT_TABLE table=default.Table_8a8adb34395948c991016903d78996cb [id=372bdce9d977439e9347e6ef616694ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.581183 889 tablet_service.cc:1467] Processing CreateTablet for tablet ecefb6f9ce75448a94a44c852b7e2cff (DEFAULT_TABLE table=default.Table_36ea26af7c544341a2862484b5a9a881 [id=8c64e6c1f93e432995dcf1ca145f8f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.581403 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 0576ffd624ac4b27b6a1c0080f940344 (DEFAULT_TABLE table=default.Table_83f317260ae141bdba6ce160f19270fe [id=871b114b3b974634ade2f7d7253b6425]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.581472 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecefb6f9ce75448a94a44c852b7e2cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.581763 1020 tablet_service.cc:1467] Processing CreateTablet for tablet ecefb6f9ce75448a94a44c852b7e2cff (DEFAULT_TABLE table=default.Table_36ea26af7c544341a2862484b5a9a881 [id=8c64e6c1f93e432995dcf1ca145f8f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.582777 912 raft_consensus.cc:2463] T 61eebc5f22664712af6e74e82809ca87 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e775296eb746db8b318f17d26820e6 in term 1.
I20250112 02:52:49.583274 989 leader_election.cc:304] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8
I20250112 02:52:49.583902 3266 raft_consensus.cc:2798] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:49.584224 3266 raft_consensus.cc:695] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Becoming Leader. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Running, Role: LEADER
I20250112 02:52:49.584580 3266 consensus_queue.cc:237] T 61eebc5f22664712af6e74e82809ca87 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.587808 3355 tablet.cc:1721] T e3ee5d8a17874d7ba54de51568e7d2e6 P fe36951a83fb4e57b26e187edbcbfe3d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:49.588011 895 tablet_service.cc:1467] Processing CreateTablet for tablet 0901eb8b296d40eba419b2a1700014f9 (DEFAULT_TABLE table=default.Table_9d1a5f1a606f4ea886dcf4616f6036d5 [id=1ff7d0e5b2b44e9eb8f04eda9e3d3068]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.588477 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0901eb8b296d40eba419b2a1700014f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.527197 1030 tablet_service.cc:1467] Processing CreateTablet for tablet e1ac5b5ea3a6494baf806bd4441e747f (DEFAULT_TABLE table=default.Table_3f00edd22a8048609d508e9a9545a361 [id=b58c92930dda4f2c882eb730024971bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.532517 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 083fcf0c7f584494832a3b05b0842dc3 (DEFAULT_TABLE table=default.Table_f5445f2c41c740a5bf3ceb49b8491359 [id=f9b9ddb521fa442eb7d60b47ed99679d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.493870 1173 tablet_bootstrap.cc:492] T 5450777104da48f1821575658fd31afd P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:49.521018 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be856e4b913e40d58a3f3ba54698cc5f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:49.589071 1173 ts_tablet_manager.cc:1397] T 5450777104da48f1821575658fd31afd P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.466s user 0.008s sys 0.004s
I20250112 02:52:49.589744 1173 raft_consensus.cc:357] T 5450777104da48f1821575658fd31afd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.590250 1173 raft_consensus.cc:383] T 5450777104da48f1821575658fd31afd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.590368 1173 raft_consensus.cc:738] T 5450777104da48f1821575658fd31afd P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.590605 1173 consensus_queue.cc:260] T 5450777104da48f1821575658fd31afd P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.591559 1173 ts_tablet_manager.cc:1428] T 5450777104da48f1821575658fd31afd P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 02:52:49.592685 1173 tablet_bootstrap.cc:492] T fc52271535df47e282bf1625c2e8e041 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:49.594076 1173 tablet_bootstrap.cc:654] T fc52271535df47e282bf1625c2e8e041 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
W20250112 02:52:49.589504 854 leader_election.cc:343] T be856e4b913e40d58a3f3ba54698cc5f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.599529 854 leader_election.cc:304] T be856e4b913e40d58a3f3ba54698cc5f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:49.599748 3000 raft_consensus.cc:2743] T be856e4b913e40d58a3f3ba54698cc5f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:49.603168 3110 ts_tablet_manager.cc:1918] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:52:49.607224 3106 raft_consensus.cc:491] T d95f98ae7bdf42518bbff55e257fea88 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.607333 3106 raft_consensus.cc:513] T d95f98ae7bdf42518bbff55e257fea88 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.607753 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67eca0da8346407fa4687738328dc414. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.608151 3106 leader_election.cc:290] T d95f98ae7bdf42518bbff55e257fea88 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:49.608675 713 leader_election.cc:343] T d95f98ae7bdf42518bbff55e257fea88 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Not found: Tablet not found: d95f98ae7bdf42518bbff55e257fea88
W20250112 02:52:49.609617 713 consensus_peers.cc:487] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5804fc4130f84c69832c1022b7bd7fa5. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:49.611598 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0576ffd624ac4b27b6a1c0080f940344. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.611979 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecefb6f9ce75448a94a44c852b7e2cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.612321 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 083fcf0c7f584494832a3b05b0842dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.612669 3111 ts_tablet_manager.cc:1918] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 02:52:49.612785 3111 log.cc:1198] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/f8c01b80fc8242d98fdfbf767212144d
I20250112 02:52:49.613128 3111 ts_tablet_manager.cc:1939] T f8c01b80fc8242d98fdfbf767212144d P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:49.617697 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 497) took 3237 ms. Trace:
I20250112 02:52:49.620893 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 7159e0046e87455e89ba35da912df7bf (DEFAULT_TABLE table=default.Table_1b6e075c7c0a4937ac65ce1bebdc1af6 [id=a3c79a54b2444cdaa8ff21596ac4a280]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.620890 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1ac5b5ea3a6494baf806bd4441e747f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.621519 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7159e0046e87455e89ba35da912df7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.623167 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 2ef34ffff8f24f819d7f1cfafdc639de
I20250112 02:52:49.620923 3109 rpcz_store.cc:276] 0112 02:52:46.379910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.379964 (+ 54us) service_pool.cc:225] Handling call
0112 02:52:49.166025 (+2786061us) ts_tablet_manager.cc:1145] Deleting tablet 3499374810fb412e938325434c35618d
0112 02:52:49.166033 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.531101 (+365068us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.549936 (+ 18835us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.617681 (+ 67745us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2785895}
I20250112 02:52:49.623603 3109 tablet_replica.cc:331] T b3c5a861a90840d0a33b780ec75bffd2 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:49.624267 3109 raft_consensus.cc:2238] T b3c5a861a90840d0a33b780ec75bffd2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:49.624418 3109 raft_consensus.cc:2267] T b3c5a861a90840d0a33b780ec75bffd2 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:49.624918 3109 ts_tablet_manager.cc:1905] T b3c5a861a90840d0a33b780ec75bffd2 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:49.624066 1177 tablet_bootstrap.cc:492] T 09900b42e8a443e680cc35c96a1dbbb7 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:49.626402 1177 ts_tablet_manager.cc:1397] T 09900b42e8a443e680cc35c96a1dbbb7 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.076s user 0.004s sys 0.000s
I20250112 02:52:49.627051 1177 raft_consensus.cc:357] T 09900b42e8a443e680cc35c96a1dbbb7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.627213 1177 raft_consensus.cc:383] T 09900b42e8a443e680cc35c96a1dbbb7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.627264 1177 raft_consensus.cc:738] T 09900b42e8a443e680cc35c96a1dbbb7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.627213 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_7c595d13b0914741a5a30ea88e4ed5e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:49.627803 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c595d13b0914741a5a30ea88e4ed5e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.627436 1177 consensus_queue.cc:260] T 09900b42e8a443e680cc35c96a1dbbb7 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.629354 1177 ts_tablet_manager.cc:1428] T 09900b42e8a443e680cc35c96a1dbbb7 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 02:52:49.629586 1177 tablet_bootstrap.cc:492] T 781a9943f705477cb6eae16df4468d2f P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:49.630038 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef34ffff8f24f819d7f1cfafdc639de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:49 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:49.630363 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0bcd74d7f041d8b84bc5fe08b18093 (DEFAULT_TABLE table=default.Table_b529da05fc294c69bca4cfa030291c41 [id=293c8b2ed5514f68a1f8c594dbabae1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.630638 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b0bcd74d7f041d8b84bc5fe08b18093. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.633149 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 1d909275d67c4127b3144de53efc7d3e
I20250112 02:52:49.633795 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d909275d67c4127b3144de53efc7d3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:49 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:49.635834 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_1169222985d54d2b843b703bec2578f4" } new_table_name: "default.Table_bec58079144546c0829d097255637176" modify_external_catalogs: true
I20250112 02:52:49.640602 1177 tablet_bootstrap.cc:654] T 781a9943f705477cb6eae16df4468d2f P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.645656 3109 ts_tablet_manager.cc:1918] T b3c5a861a90840d0a33b780ec75bffd2 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:49.647516 3310 tablet.cc:1721] T 05000c6a48e0450d8e6685ca864f67f0 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:49.647795 3310 tablet_metadata.cc:623] Not flushing: waiting for 1 pins to be released.
I20250112 02:52:49.648291 3310 tablet_bootstrap.cc:492] T 05000c6a48e0450d8e6685ca864f67f0 P f1e775296eb746db8b318f17d26820e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 02:52:49.648912 3348 tablet.cc:1721] T 2c03b5e612d64314b210c11686fdfa2a P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 8
I20250112 02:52:49.653815 3332 tablet.cc:1721] T e3ee5d8a17874d7ba54de51568e7d2e6 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:49.657325 743 tablet_service.cc:1467] Processing CreateTablet for tablet d882205901e946a6b624c7aefc6a3921 (DEFAULT_TABLE table=default.Table_211137fe64404ad5ba736c0aea830162 [id=9545de1e92df466da0b695427cb996fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.657559 3109 log.cc:1198] T b3c5a861a90840d0a33b780ec75bffd2 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/b3c5a861a90840d0a33b780ec75bffd2
I20250112 02:52:49.657774 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d882205901e946a6b624c7aefc6a3921. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.647327 1177 tablet_bootstrap.cc:492] T 781a9943f705477cb6eae16df4468d2f P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:49.658042 1177 ts_tablet_manager.cc:1397] T 781a9943f705477cb6eae16df4468d2f P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20250112 02:52:49.658895 1177 raft_consensus.cc:357] T 781a9943f705477cb6eae16df4468d2f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.659106 1177 raft_consensus.cc:383] T 781a9943f705477cb6eae16df4468d2f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.659159 1177 raft_consensus.cc:738] T 781a9943f705477cb6eae16df4468d2f P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.656749 747 tablet_service.cc:1514] Processing DeleteTablet for tablet f5492416a0544951bcbf8a4f84942e66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:49 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:49.656847 748 tablet_service.cc:1467] Processing CreateTablet for tablet 51155a89d1b6442e990d8a3de7756a87 (DEFAULT_TABLE table=default.Table_3140af67d0f7472ea77365c505900416 [id=a0d79330a98447029ef10c8d550f64c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.659559 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51155a89d1b6442e990d8a3de7756a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.663396 743 tablet_service.cc:1467] Processing CreateTablet for tablet 083fcf0c7f584494832a3b05b0842dc3 (DEFAULT_TABLE table=default.Table_f5445f2c41c740a5bf3ceb49b8491359 [id=f9b9ddb521fa442eb7d60b47ed99679d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.657308 745 tablet_service.cc:1467] Processing CreateTablet for tablet e1ac5b5ea3a6494baf806bd4441e747f (DEFAULT_TABLE table=default.Table_3f00edd22a8048609d508e9a9545a361 [id=b58c92930dda4f2c882eb730024971bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.663902 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 083fcf0c7f584494832a3b05b0842dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.664161 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1ac5b5ea3a6494baf806bd4441e747f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.670691 1177 consensus_queue.cc:260] T 781a9943f705477cb6eae16df4468d2f P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.671293 1177 ts_tablet_manager.cc:1428] T 781a9943f705477cb6eae16df4468d2f P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250112 02:52:49.671718 1177 tablet_bootstrap.cc:492] T a470f84d50df4cf0b4a1e5d248b4c32a P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:49.619233 3397 raft_consensus.cc:491] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.620079 885 tablet_service.cc:1467] Processing CreateTablet for tablet 7159e0046e87455e89ba35da912df7bf (DEFAULT_TABLE table=default.Table_1b6e075c7c0a4937ac65ce1bebdc1af6 [id=a3c79a54b2444cdaa8ff21596ac4a280]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.682428 3397 raft_consensus.cc:513] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.682597 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7159e0046e87455e89ba35da912df7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.682976 3397 leader_election.cc:290] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:49.683259 3397 raft_consensus.cc:491] T 574de2480cbb4498b414c284fea30b2c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.683351 3397 raft_consensus.cc:513] T 574de2480cbb4498b414c284fea30b2c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.683933 3397 leader_election.cc:290] T 574de2480cbb4498b414c284fea30b2c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:49.684635 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f950c604609042da96e17398f42c1929" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:49.684981 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574de2480cbb4498b414c284fea30b2c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
I20250112 02:52:49.686779 3307 tablet_bootstrap.cc:492] T 24ebf85f76f44633a20d97964638c0ac P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
I20250112 02:52:49.620196 886 tablet_service.cc:1467] Processing CreateTablet for tablet 67eca0da8346407fa4687738328dc414 (DEFAULT_TABLE table=default.Table_8a8adb34395948c991016903d78996cb [id=372bdce9d977439e9347e6ef616694ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.625337 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef34ffff8f24f819d7f1cfafdc639de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:49 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:49.687557 3397 raft_consensus.cc:491] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.687655 3397 raft_consensus.cc:513] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.687769 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67eca0da8346407fa4687738328dc414. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.688088 3397 leader_election.cc:290] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
W20250112 02:52:49.688390 855 leader_election.cc:343] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:52:49.688522 855 leader_election.cc:343] T 574de2480cbb4498b414c284fea30b2c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.691800 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b455a37448a044679bd2c0646de3d804" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:49.692370 855 leader_election.cc:343] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.626618 896 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0bcd74d7f041d8b84bc5fe08b18093 (DEFAULT_TABLE table=default.Table_b529da05fc294c69bca4cfa030291c41 [id=293c8b2ed5514f68a1f8c594dbabae1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.692827 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b0bcd74d7f041d8b84bc5fe08b18093. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.634696 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d909275d67c4127b3144de53efc7d3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:49 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:49.694774 3109 ts_tablet_manager.cc:1939] T b3c5a861a90840d0a33b780ec75bffd2 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:49.659920 893 tablet_service.cc:1467] Processing CreateTablet for tablet 0576ffd624ac4b27b6a1c0080f940344 (DEFAULT_TABLE table=default.Table_83f317260ae141bdba6ce160f19270fe [id=871b114b3b974634ade2f7d7253b6425]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.670395 1181 tablet_bootstrap.cc:492] T 6b1d46c5395d4defb2d0e2948807eaf4 P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:49.694984 1181 ts_tablet_manager.cc:1397] T 6b1d46c5395d4defb2d0e2948807eaf4 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.137s user 0.014s sys 0.018s
I20250112 02:52:49.695086 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0576ffd624ac4b27b6a1c0080f940344. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.695611 1181 raft_consensus.cc:357] T 6b1d46c5395d4defb2d0e2948807eaf4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.695766 1177 tablet_bootstrap.cc:654] T a470f84d50df4cf0b4a1e5d248b4c32a P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.695777 1181 raft_consensus.cc:383] T 6b1d46c5395d4defb2d0e2948807eaf4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.695818 1181 raft_consensus.cc:738] T 6b1d46c5395d4defb2d0e2948807eaf4 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.695991 1181 consensus_queue.cc:260] T 6b1d46c5395d4defb2d0e2948807eaf4 P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.696482 1181 ts_tablet_manager.cc:1428] T 6b1d46c5395d4defb2d0e2948807eaf4 P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 02:52:49.696697 1181 tablet_bootstrap.cc:492] T d1269a6ed80e46ecb7f4f87e5afb732f P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:49.697873 1181 tablet_bootstrap.cc:654] T d1269a6ed80e46ecb7f4f87e5afb732f P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.698407 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 498) took 3318 ms. Trace:
I20250112 02:52:49.698482 3109 rpcz_store.cc:276] 0112 02:52:46.379999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.380175 (+ 176us) service_pool.cc:225] Handling call
0112 02:52:49.623565 (+3243390us) ts_tablet_manager.cc:1145] Deleting tablet b3c5a861a90840d0a33b780ec75bffd2
0112 02:52:49.623574 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.637732 (+ 14158us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.645572 (+ 7840us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.698391 (+ 52819us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3243247}
I20250112 02:52:49.698827 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 499) took 3318 ms. Trace:
I20250112 02:52:49.691849 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 515) took 2387 ms. Trace:
I20250112 02:52:49.699304 3111 rpcz_store.cc:276] 0112 02:52:47.304736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.315138 (+ 10402us) service_pool.cc:225] Handling call
0112 02:52:49.373924 (+2058786us) ts_tablet_manager.cc:1145] Deleting tablet f8c01b80fc8242d98fdfbf767212144d
0112 02:52:49.373937 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.544387 (+170450us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.612603 (+ 68216us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.691833 (+ 79230us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2048949}
I20250112 02:52:49.700564 3111 tablet_replica.cc:331] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:49.703447 3111 raft_consensus.cc:2238] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:49.698900 3109 rpcz_store.cc:276] 0112 02:52:46.380117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.380345 (+ 228us) service_pool.cc:225] Handling call
0112 02:52:49.698738 (+3318393us) ts_tablet_manager.cc:1145] Deleting tablet 61d7703b15024fe7a35805345d5e81ab
0112 02:52:49.698747 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.698815 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3318269}
I20250112 02:52:49.706799 3109 tablet_replica.cc:331] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:49.706972 3109 raft_consensus.cc:2238] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:49.707244 3109 raft_consensus.cc:2267] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:49.707733 3109 ts_tablet_manager.cc:1905] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:49.709820 1173 tablet_bootstrap.cc:492] T fc52271535df47e282bf1625c2e8e041 P f1e775296eb746db8b318f17d26820e6: No bootstrap required, opened a new log
I20250112 02:52:49.710489 1173 ts_tablet_manager.cc:1397] T fc52271535df47e282bf1625c2e8e041 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.118s user 0.003s sys 0.000s
I20250112 02:52:49.710251 3111 raft_consensus.cc:2267] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:49.711416 3111 ts_tablet_manager.cc:1905] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:49.715764 1173 raft_consensus.cc:357] T fc52271535df47e282bf1625c2e8e041 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.720985 3110 log.cc:1198] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/a1123077650c4e89bffddd6d1264e428
I20250112 02:52:49.721756 753 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0bcd74d7f041d8b84bc5fe08b18093 (DEFAULT_TABLE table=default.Table_b529da05fc294c69bca4cfa030291c41 [id=293c8b2ed5514f68a1f8c594dbabae1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.722079 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b0bcd74d7f041d8b84bc5fe08b18093. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.722430 3110 ts_tablet_manager.cc:1939] T a1123077650c4e89bffddd6d1264e428 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:49.723587 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d909275d67c4127b3144de53efc7d3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:49 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:49.724220 760 tablet_service.cc:1467] Processing CreateTablet for tablet 7159e0046e87455e89ba35da912df7bf (DEFAULT_TABLE table=default.Table_1b6e075c7c0a4937ac65ce1bebdc1af6 [id=a3c79a54b2444cdaa8ff21596ac4a280]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.724602 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7159e0046e87455e89ba35da912df7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.727547 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef34ffff8f24f819d7f1cfafdc639de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:49 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:49.727684 747 tablet_service.cc:1467] Processing CreateTablet for tablet ecefb6f9ce75448a94a44c852b7e2cff (DEFAULT_TABLE table=default.Table_36ea26af7c544341a2862484b5a9a881 [id=8c64e6c1f93e432995dcf1ca145f8f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.727958 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecefb6f9ce75448a94a44c852b7e2cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.731506 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 398) took 3186 ms. Trace:
I20250112 02:52:49.731590 3110 rpcz_store.cc:276] 0112 02:52:46.545049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.545111 (+ 62us) service_pool.cc:225] Handling call
0112 02:52:49.321376 (+2776265us) ts_tablet_manager.cc:1145] Deleting tablet a1123077650c4e89bffddd6d1264e428
0112 02:52:49.421018 (+ 99642us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.507112 (+ 86094us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.603100 (+ 95988us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.731488 (+128388us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2776075}
I20250112 02:52:49.731998 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet a1123077650c4e89bffddd6d1264e428 (table default.Table_36b06d2edb8e4d23a3cb7021a8267db4 [id=3506f201d2674e38a353c87e1b2f26a5]) successfully deleted
I20250112 02:52:49.735189 745 tablet_service.cc:1467] Processing CreateTablet for tablet 67eca0da8346407fa4687738328dc414 (DEFAULT_TABLE table=default.Table_8a8adb34395948c991016903d78996cb [id=372bdce9d977439e9347e6ef616694ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.735342 896 tablet_service.cc:1467] Processing CreateTablet for tablet 1db4f8b5532c4692968309fb803fb531 (DEFAULT_TABLE table=default.Table_ee4d7e840acc457c88cd3b4de3824bbe [id=632e835766a04e5bbb3d19c621434768]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.735571 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67eca0da8346407fa4687738328dc414. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.735739 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1db4f8b5532c4692968309fb803fb531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.739568 748 tablet_service.cc:1467] Processing CreateTablet for tablet 0576ffd624ac4b27b6a1c0080f940344 (DEFAULT_TABLE table=default.Table_83f317260ae141bdba6ce160f19270fe [id=871b114b3b974634ade2f7d7253b6425]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.739933 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0576ffd624ac4b27b6a1c0080f940344. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.740701 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 3499374810fb412e938325434c35618d (table default.Table_7870d93893b74b4b96d1a3809a9015e6 [id=4355589b5ccc4162ab8700e31173d1e5]) successfully deleted
I20250112 02:52:49.741518 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet b3c5a861a90840d0a33b780ec75bffd2 (table default.Table_c26869f6cbd540fe980dcf207b5c729d [id=cdd6a2097792484d84bca856f30d7ac1]) successfully deleted
W20250112 02:52:49.741748 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 61d7703b15024fe7a35805345d5e81ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61d7703b15024fe7a35805345d5e81ab already in progress: creating tablet
I20250112 02:52:49.743525 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_903b8aaa85524d2fa1c614b794991a25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:49.744100 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_903b8aaa85524d2fa1c614b794991a25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.748523 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_65742cc1c2824757830aec593a96efd8" } new_table_name: "default.Table_857f92d093b24cf28fa981b8cd0bb049" modify_external_catalogs: true
I20250112 02:52:49.717382 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574de2480cbb4498b414c284fea30b2c" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:49.718083 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b455a37448a044679bd2c0646de3d804" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:49.718231 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f950c604609042da96e17398f42c1929" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:49.751987 854 leader_election.cc:343] T 574de2480cbb4498b414c284fea30b2c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.733955 3310 tablet_bootstrap.cc:492] T 05000c6a48e0450d8e6685ca864f67f0 P f1e775296eb746db8b318f17d26820e6: Bootstrap complete.
I20250112 02:52:49.744634 3307 tablet.cc:1721] T 24ebf85f76f44633a20d97964638c0ac P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:52:49.752079 1173 raft_consensus.cc:383] T fc52271535df47e282bf1625c2e8e041 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.752211 3307 tablet_metadata.cc:623] Not flushing: waiting for 1 pins to be released.
I20250112 02:52:49.752513 3307 tablet_bootstrap.cc:492] T 24ebf85f76f44633a20d97964638c0ac P f1e775296eb746db8b318f17d26820e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 02:52:49.752796 3307 tablet_bootstrap.cc:492] T 24ebf85f76f44633a20d97964638c0ac P f1e775296eb746db8b318f17d26820e6: Bootstrap complete.
I20250112 02:52:49.759799 3109 ts_tablet_manager.cc:1918] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:49.759945 3109 log.cc:1198] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/5804fc4130f84c69832c1022b7bd7fa5
I20250112 02:52:49.760666 3307 ts_tablet_manager.cc:1397] T 24ebf85f76f44633a20d97964638c0ac P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.074s user 0.008s sys 0.003s
I20250112 02:52:49.760800 3332 tablet.cc:1721] T 47a0d939fdb94590a86389f1f4693e9a P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:49.761548 3109 ts_tablet_manager.cc:1939] T 5804fc4130f84c69832c1022b7bd7fa5 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:49.761729 3374 tablet.cc:1721] T 47a0d939fdb94590a86389f1f4693e9a P 289afc116aa74408aa5bff74e11c50d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:52:49.761595 3307 raft_consensus.cc:357] T 24ebf85f76f44633a20d97964638c0ac P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.761865 3307 raft_consensus.cc:738] T 24ebf85f76f44633a20d97964638c0ac P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.762051 3307 consensus_queue.cc:260] T 24ebf85f76f44633a20d97964638c0ac P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.762550 3307 ts_tablet_manager.cc:1428] T 24ebf85f76f44633a20d97964638c0ac P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 02:52:49.763933 800 tablet_copy_service.cc:342] P 289afc116aa74408aa5bff74e11c50d8: Request end of tablet copy session f1e775296eb746db8b318f17d26820e6-24ebf85f76f44633a20d97964638c0ac received from {username='slave'} at 127.0.104.3:52505
I20250112 02:52:49.764045 800 tablet_copy_service.cc:434] P 289afc116aa74408aa5bff74e11c50d8: ending tablet copy session f1e775296eb746db8b318f17d26820e6-24ebf85f76f44633a20d97964638c0ac on tablet 24ebf85f76f44633a20d97964638c0ac with peer f1e775296eb746db8b318f17d26820e6
I20250112 02:52:49.766865 3111 ts_tablet_manager.cc:1918] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 02:52:49.766944 3111 log.cc:1198] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/5d0c3191973a477e949fbf9dfcca963f
I20250112 02:52:49.767256 3111 ts_tablet_manager.cc:1939] T 5d0c3191973a477e949fbf9dfcca963f P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:49.767879 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 516) took 2443 ms. Trace:
I20250112 02:52:49.767951 3111 rpcz_store.cc:276] 0112 02:52:47.324459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.325577 (+ 1118us) service_pool.cc:225] Handling call
0112 02:52:49.700520 (+2374943us) ts_tablet_manager.cc:1145] Deleting tablet 5d0c3191973a477e949fbf9dfcca963f
0112 02:52:49.700531 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.714367 (+ 13836us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.766787 (+ 52420us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.767862 (+ 1075us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6496,"tablet-delete.queue_time_us":2374772}
I20250112 02:52:49.768357 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 538) took 2428 ms. Trace:
I20250112 02:52:49.768443 3111 rpcz_store.cc:276] 0112 02:52:47.340067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.345367 (+ 5300us) service_pool.cc:225] Handling call
0112 02:52:49.768199 (+2422832us) ts_tablet_manager.cc:1145] Deleting tablet bf2df39addb14d679b8cb7960a62d603
0112 02:52:49.768226 (+ 27us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.768342 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2422695}
I20250112 02:52:49.768656 3111 tablet_replica.cc:331] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:49.768821 3111 raft_consensus.cc:2238] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:49.768955 3111 raft_consensus.cc:2267] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:49.769475 3111 ts_tablet_manager.cc:1905] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:49.769454 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 5d0c3191973a477e949fbf9dfcca963f (table default.Table_9b25aff63ebb4620a3efe624832af86c [id=2f43ebd749304c0287769224ac6b12c7]) successfully deleted
W20250112 02:52:49.769716 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet bf2df39addb14d679b8cb7960a62d603 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf2df39addb14d679b8cb7960a62d603 already in progress: creating tablet
I20250112 02:52:49.774933 3310 ts_tablet_manager.cc:1397] T 05000c6a48e0450d8e6685ca864f67f0 P f1e775296eb746db8b318f17d26820e6: Time spent bootstrapping tablet: real 0.428s user 0.009s sys 0.000s
I20250112 02:52:49.775070 854 leader_election.cc:304] T 574de2480cbb4498b414c284fea30b2c P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:49.775377 3397 raft_consensus.cc:2743] T 574de2480cbb4498b414c284fea30b2c P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:49.775904 854 leader_election.cc:343] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.776049 854 leader_election.cc:304] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:49.776247 3397 raft_consensus.cc:2743] T b455a37448a044679bd2c0646de3d804 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:52:49.776660 854 leader_election.cc:343] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.776816 854 leader_election.cc:304] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe36951a83fb4e57b26e187edbcbfe3d; no voters: 289afc116aa74408aa5bff74e11c50d8, f1e775296eb746db8b318f17d26820e6
I20250112 02:52:49.777029 3397 raft_consensus.cc:2743] T f950c604609042da96e17398f42c1929 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:49.778877 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 399) took 3202 ms. Trace:
I20250112 02:52:49.778990 3110 rpcz_store.cc:276] 0112 02:52:46.576320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.576380 (+ 60us) service_pool.cc:225] Handling call
0112 02:52:49.731837 (+3155457us) ts_tablet_manager.cc:1145] Deleting tablet f5c671e327a148ecb1d08a983448946e
0112 02:52:49.731844 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.778864 (+ 47020us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3155263}
W20250112 02:52:49.779561 440 catalog_manager.cc:4852] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): delete failed for tablet f5c671e327a148ecb1d08a983448946e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5c671e327a148ecb1d08a983448946e already in progress: creating tablet
I20250112 02:52:49.779946 3110 tablet_replica.cc:331] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:49.780210 3110 raft_consensus.cc:2238] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:49.780442 3110 raft_consensus.cc:2267] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:49.781090 3110 ts_tablet_manager.cc:1905] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:49.783785 743 tablet_service.cc:1467] Processing CreateTablet for tablet 6506fb4f1dee4bd09f0f3cf52c6c1b2e (DEFAULT_TABLE table=default.Table_6b747ebd44244031a16f8b805caa9209 [id=6d83348798094235be56595652dccd6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.784312 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6506fb4f1dee4bd09f0f3cf52c6c1b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.787053 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: Sending DeleteTablet for 3 replicas of tablet 6b1d46c5395d4defb2d0e2948807eaf4
I20250112 02:52:49.787746 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_99896ca24c1a4eb1a31b036b021a8621" } modify_external_catalogs: true
I20250112 02:52:49.787930 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40610:
table { table_name: "default.Table_99896ca24c1a4eb1a31b036b021a8621" } modify_external_catalogs: true
I20250112 02:52:49.796094 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 496) took 3416 ms. Trace:
I20250112 02:52:49.796185 3109 rpcz_store.cc:276] 0112 02:52:46.379780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.380437 (+ 657us) service_pool.cc:225] Handling call
0112 02:52:49.706762 (+3326325us) ts_tablet_manager.cc:1145] Deleting tablet 5804fc4130f84c69832c1022b7bd7fa5
0112 02:52:49.706770 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.757977 (+ 51207us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.759434 (+ 1457us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.796077 (+ 36643us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33415,"tablet-delete.queue_time_us":3326178}
I20250112 02:52:49.796492 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 500) took 3416 ms. Trace:
I20250112 02:52:49.796561 3109 rpcz_store.cc:276] 0112 02:52:46.380261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.382583 (+ 2322us) service_pool.cc:225] Handling call
0112 02:52:49.796415 (+3413832us) ts_tablet_manager.cc:1145] Deleting tablet f776e2a6ec704cf9b64483bc7d5b1dff
0112 02:52:49.796423 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.796481 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3413000}
I20250112 02:52:49.796782 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 507) took 3253 ms. Trace:
I20250112 02:52:49.796828 3109 rpcz_store.cc:276] 0112 02:52:46.543051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.543115 (+ 64us) service_pool.cc:225] Handling call
0112 02:52:49.796709 (+3253594us) ts_tablet_manager.cc:1145] Deleting tablet a1123077650c4e89bffddd6d1264e428
0112 02:52:49.796715 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.796772 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3214420}
I20250112 02:52:49.797050 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 508) took 3230 ms. Trace:
I20250112 02:52:49.797102 3109 rpcz_store.cc:276] 0112 02:52:46.566426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.567853 (+ 1427us) service_pool.cc:225] Handling call
0112 02:52:49.796975 (+3229122us) ts_tablet_manager.cc:1145] Deleting tablet f5c671e327a148ecb1d08a983448946e
0112 02:52:49.796983 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.797040 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3213983}
I20250112 02:52:49.797319 3109 tablet_replica.cc:331] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:49.797478 3109 raft_consensus.cc:2238] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:49.797593 3109 raft_consensus.cc:2267] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:49.798045 3109 ts_tablet_manager.cc:1905] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:49.800076 3138 raft_consensus.cc:491] T 09900b42e8a443e680cc35c96a1dbbb7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.800211 3138 raft_consensus.cc:513] T 09900b42e8a443e680cc35c96a1dbbb7 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.800885 3138 leader_election.cc:290] T 09900b42e8a443e680cc35c96a1dbbb7 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:49.801720 3133 raft_consensus.cc:491] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.801839 3133 raft_consensus.cc:513] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.802395 3133 leader_election.cc:290] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:49.808419 3109 ts_tablet_manager.cc:1918] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:49.808554 3109 log.cc:1198] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/e477fc157aed4563923d85d66014293a
I20250112 02:52:49.808912 3109 ts_tablet_manager.cc:1939] T e477fc157aed4563923d85d66014293a P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:49.809631 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 510) took 3156 ms. Trace:
I20250112 02:52:49.809757 3109 rpcz_store.cc:276] 0112 02:52:46.652684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.652793 (+ 109us) service_pool.cc:225] Handling call
0112 02:52:49.797261 (+3144468us) ts_tablet_manager.cc:1145] Deleting tablet e477fc157aed4563923d85d66014293a
0112 02:52:49.797276 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.805841 (+ 8565us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.808339 (+ 2498us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.809613 (+ 1274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3137984}
I20250112 02:52:49.810168 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 515) took 3088 ms. Trace:
I20250112 02:52:49.810287 3109 rpcz_store.cc:276] 0112 02:52:46.721879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.721943 (+ 64us) service_pool.cc:225] Handling call
0112 02:52:49.810062 (+3088119us) ts_tablet_manager.cc:1145] Deleting tablet 8c12a1c5e3584450ac193743a0107e80
0112 02:52:49.810071 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.810142 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3035106}
I20250112 02:52:49.810612 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 520) took 3022 ms. Trace:
I20250112 02:52:49.810746 3109 rpcz_store.cc:276] 0112 02:52:46.788253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.788305 (+ 52us) service_pool.cc:225] Handling call
0112 02:52:49.810539 (+3022234us) ts_tablet_manager.cc:1145] Deleting tablet 13578542e9e741e8817e0ee63b10f1a1
0112 02:52:49.810548 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.810598 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3021865}
I20250112 02:52:49.811165 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 518) took 3036 ms. Trace:
I20250112 02:52:49.811293 3109 rpcz_store.cc:276] 0112 02:52:46.774979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.775028 (+ 49us) service_pool.cc:225] Handling call
0112 02:52:49.811051 (+3036023us) ts_tablet_manager.cc:1145] Deleting tablet 99407ead9cf547e3bbe78c0d0b327e6a
0112 02:52:49.811061 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.811150 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3021964}
I20250112 02:52:49.811725 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 516) took 3069 ms. Trace:
I20250112 02:52:49.811844 3109 rpcz_store.cc:276] 0112 02:52:46.741809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.741955 (+ 146us) service_pool.cc:225] Handling call
0112 02:52:49.811590 (+3069635us) ts_tablet_manager.cc:1145] Deleting tablet 17c54c5dc86446cf85974777c8145ef6
0112 02:52:49.811600 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.811710 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3022464}
I20250112 02:52:49.812170 3109 tablet_replica.cc:331] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8: stopping tablet replica
I20250112 02:52:49.812431 3109 raft_consensus.cc:2238] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:49.812830 3109 pending_rounds.cc:62] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8: Trying to abort 1 pending ops.
I20250112 02:52:49.812925 3109 pending_rounds.cc:69] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113319908470460416 op_type: NO_OP noop_request { }
I20250112 02:52:49.813091 3109 raft_consensus.cc:2883] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:52:49.813230 3109 raft_consensus.cc:2267] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:49.813792 3109 ts_tablet_manager.cc:1905] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:49.826138 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet 5804fc4130f84c69832c1022b7bd7fa5 (table default.Table_4001552c6dc246e18a8b0ad9041267c6 [id=370e66eb490a48b5927d073f94481de0]) successfully deleted
W20250112 02:52:49.826503 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f776e2a6ec704cf9b64483bc7d5b1dff because the tablet was not found. No further retry: Not found: Tablet not found: f776e2a6ec704cf9b64483bc7d5b1dff
W20250112 02:52:49.826871 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet a1123077650c4e89bffddd6d1264e428 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1123077650c4e89bffddd6d1264e428 already in progress: creating tablet
W20250112 02:52:49.827384 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet f5c671e327a148ecb1d08a983448946e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5c671e327a148ecb1d08a983448946e already in progress: creating tablet
I20250112 02:52:49.827966 442 catalog_manager.cc:4872] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): tablet e477fc157aed4563923d85d66014293a (table default.Table_e495ee84abe3479c80c8ab2e14f86c96 [id=f5eb0a253e444d76978a22250c163a54]) successfully deleted
W20250112 02:52:49.828282 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 8c12a1c5e3584450ac193743a0107e80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c12a1c5e3584450ac193743a0107e80 already in progress: creating tablet
W20250112 02:52:49.828613 442 catalog_manager.cc:4839] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 13578542e9e741e8817e0ee63b10f1a1 because the tablet was not found. No further retry: Not found: Tablet not found: 13578542e9e741e8817e0ee63b10f1a1
W20250112 02:52:49.828914 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 99407ead9cf547e3bbe78c0d0b327e6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99407ead9cf547e3bbe78c0d0b327e6a already in progress: creating tablet
W20250112 02:52:49.829339 442 catalog_manager.cc:4852] TS 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): delete failed for tablet 17c54c5dc86446cf85974777c8145ef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17c54c5dc86446cf85974777c8145ef6 already in progress: creating tablet
I20250112 02:52:49.820724 1177 tablet_bootstrap.cc:492] T a470f84d50df4cf0b4a1e5d248b4c32a P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:49.843613 1177 ts_tablet_manager.cc:1397] T a470f84d50df4cf0b4a1e5d248b4c32a P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.172s user 0.014s sys 0.000s
I20250112 02:52:49.844226 1177 raft_consensus.cc:357] T a470f84d50df4cf0b4a1e5d248b4c32a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.844401 1177 raft_consensus.cc:383] T a470f84d50df4cf0b4a1e5d248b4c32a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.844445 1177 raft_consensus.cc:738] T a470f84d50df4cf0b4a1e5d248b4c32a P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.844596 1177 consensus_queue.cc:260] T a470f84d50df4cf0b4a1e5d248b4c32a P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.845044 1177 ts_tablet_manager.cc:1428] T a470f84d50df4cf0b4a1e5d248b4c32a P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 02:52:49.845273 1177 tablet_bootstrap.cc:492] T 61eebc5f22664712af6e74e82809ca87 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:49.846441 1177 tablet_bootstrap.cc:654] T 61eebc5f22664712af6e74e82809ca87 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.830307 748 tablet_service.cc:1467] Processing CreateTablet for tablet 1db4f8b5532c4692968309fb803fb531 (DEFAULT_TABLE table=default.Table_ee4d7e840acc457c88cd3b4de3824bbe [id=632e835766a04e5bbb3d19c621434768]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.847131 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1db4f8b5532c4692968309fb803fb531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.848515 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_8319108d54a64cb9a768ad231cd08ae8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:52:49.849188 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8319108d54a64cb9a768ad231cd08ae8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.830595 747 tablet_service.cc:1467] Processing CreateTablet for tablet d9cd39c86b324450be0e477fd6c2496f (DEFAULT_TABLE table=default.Table_f2259d9cdec345a1923599bd9b188e92 [id=1e228766a3b148e082e16f55e97e1862]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.850332 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9cd39c86b324450be0e477fd6c2496f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.830816 745 tablet_service.cc:1467] Processing CreateTablet for tablet 878376c9593a4246a92de0a2fe42aca8 (DEFAULT_TABLE table=default.Table_5ad4c4df7d3944b898099c4681510214 [id=52ccea9492db4ed09ba48d53b0d0c0ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.855509 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 878376c9593a4246a92de0a2fe42aca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.782744 896 tablet_service.cc:1467] Processing CreateTablet for tablet d9cd39c86b324450be0e477fd6c2496f (DEFAULT_TABLE table=default.Table_f2259d9cdec345a1923599bd9b188e92 [id=1e228766a3b148e082e16f55e97e1862]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.860366 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9cd39c86b324450be0e477fd6c2496f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.830978 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b1d46c5395d4defb2d0e2948807eaf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:49 UTC) from {username='slave'} at 127.0.0.1:49624
I20250112 02:52:49.789356 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b1d46c5395d4defb2d0e2948807eaf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:49 UTC) from {username='slave'} at 127.0.0.1:38786
I20250112 02:52:49.803114 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09900b42e8a443e680cc35c96a1dbbb7" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:49.803139 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc52271535df47e282bf1625c2e8e041" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" is_pre_election: true
I20250112 02:52:49.868163 916 raft_consensus.cc:2463] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 0.
I20250112 02:52:49.868630 713 leader_election.cc:304] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:49.868876 3106 raft_consensus.cc:2798] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:52:49.868928 3106 raft_consensus.cc:491] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:52:49.868975 3106 raft_consensus.cc:3054] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:52:49.869670 3292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:52:49.870021 3292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:52:49.870143 3292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250112 02:52:49.871227 3106 raft_consensus.cc:513] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.816141 3397 raft_consensus.cc:491] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.871605 3397 raft_consensus.cc:513] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.872169 3397 leader_election.cc:290] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:49.816346 1181 tablet_bootstrap.cc:492] T d1269a6ed80e46ecb7f4f87e5afb732f P fe36951a83fb4e57b26e187edbcbfe3d: No bootstrap required, opened a new log
I20250112 02:52:49.872378 1181 ts_tablet_manager.cc:1397] T d1269a6ed80e46ecb7f4f87e5afb732f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent bootstrapping tablet: real 0.176s user 0.009s sys 0.027s
I20250112 02:52:49.872915 1181 raft_consensus.cc:357] T d1269a6ed80e46ecb7f4f87e5afb732f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.873063 1181 raft_consensus.cc:383] T d1269a6ed80e46ecb7f4f87e5afb732f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.873099 1181 raft_consensus.cc:738] T d1269a6ed80e46ecb7f4f87e5afb732f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe36951a83fb4e57b26e187edbcbfe3d, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.873252 1181 consensus_queue.cc:260] T d1269a6ed80e46ecb7f4f87e5afb732f P fe36951a83fb4e57b26e187edbcbfe3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.873673 1181 ts_tablet_manager.cc:1428] T d1269a6ed80e46ecb7f4f87e5afb732f P fe36951a83fb4e57b26e187edbcbfe3d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250112 02:52:49.873792 1177 tablet_bootstrap.cc:492] T 61eebc5f22664712af6e74e82809ca87 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:49.873910 1177 ts_tablet_manager.cc:1397] T 61eebc5f22664712af6e74e82809ca87 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.001s
I20250112 02:52:49.873960 1181 tablet_bootstrap.cc:492] T ab183c43f6a64270ad744e61364f0a99 P fe36951a83fb4e57b26e187edbcbfe3d: Bootstrap starting.
I20250112 02:52:49.875177 1181 tablet_bootstrap.cc:654] T ab183c43f6a64270ad744e61364f0a99 P fe36951a83fb4e57b26e187edbcbfe3d: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.875370 1177 raft_consensus.cc:357] T 61eebc5f22664712af6e74e82809ca87 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.875538 1177 raft_consensus.cc:383] T 61eebc5f22664712af6e74e82809ca87 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.875600 1177 raft_consensus.cc:738] T 61eebc5f22664712af6e74e82809ca87 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.876044 1177 consensus_queue.cc:260] T 61eebc5f22664712af6e74e82809ca87 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.876567 1177 ts_tablet_manager.cc:1428] T 61eebc5f22664712af6e74e82809ca87 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 02:52:49.876767 1177 tablet_bootstrap.cc:492] T 54d3664071b146a3ac7c6885f9bda789 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:49.877351 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f559dedfb7854cf890bef362c50b4213" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:49.877894 855 leader_election.cc:343] T f559dedfb7854cf890bef362c50b4213 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.885231 3106 leader_election.cc:290] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Requested vote from peers fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:49.885704 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc52271535df47e282bf1625c2e8e041" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe36951a83fb4e57b26e187edbcbfe3d"
I20250112 02:52:49.885847 916 raft_consensus.cc:3054] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:49.888120 916 raft_consensus.cc:2463] T fc52271535df47e282bf1625c2e8e041 P fe36951a83fb4e57b26e187edbcbfe3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:49.888545 713 leader_election.cc:304] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8, fe36951a83fb4e57b26e187edbcbfe3d; no voters:
I20250112 02:52:49.888613 1177 tablet_bootstrap.cc:654] T 54d3664071b146a3ac7c6885f9bda789 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.888988 3138 raft_consensus.cc:2798] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:52:49.889103 3138 raft_consensus.cc:695] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8 [term 1 LEADER]: Becoming Leader. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Running, Role: LEADER
I20250112 02:52:49.889290 3138 consensus_queue.cc:237] T fc52271535df47e282bf1625c2e8e041 P 289afc116aa74408aa5bff74e11c50d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
W20250112 02:52:49.893924 3292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 02:52:49.894311 3292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:52:49.894435 3292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 02:52:49.894551 3292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 02:52:49.782258 1022 tablet_service.cc:1467] Processing CreateTablet for tablet d9cd39c86b324450be0e477fd6c2496f (DEFAULT_TABLE table=default.Table_f2259d9cdec345a1923599bd9b188e92 [id=1e228766a3b148e082e16f55e97e1862]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.895772 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9cd39c86b324450be0e477fd6c2496f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.787739 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b1d46c5395d4defb2d0e2948807eaf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:52:49 UTC) from {username='slave'} at 127.0.0.1:56348
I20250112 02:52:49.795073 3111 ts_tablet_manager.cc:1918] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:52:49.805111 3266 consensus_queue.cc:1035] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:52:49.899538 3111 log.cc:1198] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/b8b64e4e64cc462891f1ee7a594d5026
I20250112 02:52:49.899916 3111 ts_tablet_manager.cc:1939] T b8b64e4e64cc462891f1ee7a594d5026 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:49.900656 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 540) took 2560 ms. Trace:
I20250112 02:52:49.900748 3111 rpcz_store.cc:276] 0112 02:52:47.340237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.345784 (+ 5547us) service_pool.cc:225] Handling call
0112 02:52:49.768621 (+2422837us) ts_tablet_manager.cc:1145] Deleting tablet b8b64e4e64cc462891f1ee7a594d5026
0112 02:52:49.768629 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.792387 (+ 23758us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.795006 (+ 2619us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.900639 (+105633us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2422701}
I20250112 02:52:49.901060 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 535) took 2561 ms. Trace:
I20250112 02:52:49.901125 3111 rpcz_store.cc:276] 0112 02:52:47.339827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.346146 (+ 6319us) service_pool.cc:225] Handling call
0112 02:52:49.900965 (+2554819us) ts_tablet_manager.cc:1145] Deleting tablet 4af0d07a878c42f5a15ff9b8a2295f30
0112 02:52:49.900972 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.901050 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2554690}
I20250112 02:52:49.901448 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 537) took 2561 ms. Trace:
I20250112 02:52:49.901507 3111 rpcz_store.cc:276] 0112 02:52:47.339987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.346633 (+ 6646us) service_pool.cc:225] Handling call
0112 02:52:49.901371 (+2554738us) ts_tablet_manager.cc:1145] Deleting tablet af689c1a3e2d41beb2199bf5d033aa29
0112 02:52:49.901378 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.901437 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2554585}
I20250112 02:52:49.901726 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 518) took 2563 ms. Trace:
I20250112 02:52:49.901782 3111 rpcz_store.cc:276] 0112 02:52:47.338190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.338229 (+ 39us) service_pool.cc:225] Handling call
0112 02:52:49.901658 (+2563429us) ts_tablet_manager.cc:1145] Deleting tablet 0fea8f6f13d54945934ee3880e00c5e9
0112 02:52:49.901664 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.901716 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2554442}
I20250112 02:52:49.902091 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 541) took 2323 ms. Trace:
I20250112 02:52:49.902176 3111 rpcz_store.cc:276] 0112 02:52:47.578701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.578903 (+ 202us) service_pool.cc:225] Handling call
0112 02:52:49.901979 (+2323076us) ts_tablet_manager.cc:1145] Deleting tablet 9af4a0d53030406a968b81ca5989303f
0112 02:52:49.902006 (+ 27us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.902079 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2322800}
I20250112 02:52:49.902472 3111 tablet_replica.cc:331] T 500f0c5c76e54cd8808e6c8611e99d83 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:49.905828 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet b8b64e4e64cc462891f1ee7a594d5026 (table default.Table_ce0ccdae99624243bdac1499da1790ee [id=f7f2de9507524d7cb09e1b5c54869062]) successfully deleted
W20250112 02:52:49.906167 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 4af0d07a878c42f5a15ff9b8a2295f30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4af0d07a878c42f5a15ff9b8a2295f30 already in progress: creating tablet
W20250112 02:52:49.906404 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet af689c1a3e2d41beb2199bf5d033aa29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af689c1a3e2d41beb2199bf5d033aa29 already in progress: creating tablet
W20250112 02:52:49.906623 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 0fea8f6f13d54945934ee3880e00c5e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fea8f6f13d54945934ee3880e00c5e9 already in progress: creating tablet
W20250112 02:52:49.906831 441 catalog_manager.cc:4852] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): delete failed for tablet 9af4a0d53030406a968b81ca5989303f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9af4a0d53030406a968b81ca5989303f already in progress: creating tablet
I20250112 02:52:49.908922 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet 61eebc5f22664712af6e74e82809ca87 (table default.Table_1506fd71c54a4312b1c439a2effa4c69 [id=e53d0b4d7340456896f1acdcc50db840]). Expected version 3 got 0
I20250112 02:52:49.909222 451 catalog_manager.cc:5526] T 42eed0a9f8354e5391ec2463dafdb683 P f1e775296eb746db8b318f17d26820e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e775296eb746db8b318f17d26820e6 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "f1e775296eb746db8b318f17d26820e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } health_report { overall_health: HEALTHY } } }
I20250112 02:52:49.909754 3111 raft_consensus.cc:2238] T 500f0c5c76e54cd8808e6c8611e99d83 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:49.909960 3111 raft_consensus.cc:2267] T 500f0c5c76e54cd8808e6c8611e99d83 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:49.910128 451 catalog_manager.cc:5618] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831) does not have the latest schema for tablet b7340a7a916c4b2fab32e0ba2d0ce322 (table default.Table_a23515295db7425e92ddbf1dcb7ace36 [id=824805353d914188ba250e8624065190]). Expected version 2 got 0
I20250112 02:52:49.910704 888 tablet_service.cc:1467] Processing CreateTablet for tablet 6ff7a7cc250c446db0985ffc1bcc2f91 (DEFAULT_TABLE table=default.Table_17388940cd444538a523e34ad3cf51f1 [id=371bc05048924bd3a59e7a1057c07384]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.910837 3111 ts_tablet_manager.cc:1905] T 500f0c5c76e54cd8808e6c8611e99d83 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:49.911195 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ff7a7cc250c446db0985ffc1bcc2f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.917506 3111 ts_tablet_manager.cc:1918] T 500f0c5c76e54cd8808e6c8611e99d83 P f1e775296eb746db8b318f17d26820e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:49.917615 3111 log.cc:1198] T 500f0c5c76e54cd8808e6c8611e99d83 P f1e775296eb746db8b318f17d26820e6: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-2/wal/wals/500f0c5c76e54cd8808e6c8611e99d83
I20250112 02:52:49.917970 3111 ts_tablet_manager.cc:1939] T 500f0c5c76e54cd8808e6c8611e99d83 P f1e775296eb746db8b318f17d26820e6: Deleting consensus metadata
I20250112 02:52:49.918742 3111 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56348 (request call id 545) took 2272 ms. Trace:
I20250112 02:52:49.918843 3111 rpcz_store.cc:276] 0112 02:52:47.646044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:47.646757 (+ 713us) service_pool.cc:225] Handling call
0112 02:52:49.902412 (+2255655us) ts_tablet_manager.cc:1145] Deleting tablet 500f0c5c76e54cd8808e6c8611e99d83
0112 02:52:49.902421 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.914128 (+ 11707us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.917422 (+ 3294us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.918723 (+ 1301us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7018,"tablet-delete.queue_time_us":2255485}
I20250112 02:52:49.919178 3111 tablet_replica.cc:331] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6: stopping tablet replica
I20250112 02:52:49.919342 441 catalog_manager.cc:4872] TS f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): tablet 500f0c5c76e54cd8808e6c8611e99d83 (table default.Table_67e40ef8ee824430b5d090ddc9cc03c8 [id=5c61d80dfd2746f591c7296315935aaf]) successfully deleted
I20250112 02:52:49.919390 3111 raft_consensus.cc:2238] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:52:49.922063 3310 raft_consensus.cc:357] T 05000c6a48e0450d8e6685ca864f67f0 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.922300 3310 raft_consensus.cc:738] T 05000c6a48e0450d8e6685ca864f67f0 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.922461 3310 consensus_queue.cc:260] T 05000c6a48e0450d8e6685ca864f67f0 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.922928 3310 ts_tablet_manager.cc:1428] T 05000c6a48e0450d8e6685ca864f67f0 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 02:52:49.923550 800 tablet_copy_service.cc:342] P 289afc116aa74408aa5bff74e11c50d8: Request end of tablet copy session f1e775296eb746db8b318f17d26820e6-05000c6a48e0450d8e6685ca864f67f0 received from {username='slave'} at 127.0.104.3:52505
I20250112 02:52:49.923647 800 tablet_copy_service.cc:434] P 289afc116aa74408aa5bff74e11c50d8: ending tablet copy session f1e775296eb746db8b318f17d26820e6-05000c6a48e0450d8e6685ca864f67f0 on tablet 05000c6a48e0450d8e6685ca864f67f0 with peer f1e775296eb746db8b318f17d26820e6
I20250112 02:52:49.924675 748 tablet_service.cc:1467] Processing CreateTablet for tablet 27752da73ed441629754b812e1fcf4a9 (DEFAULT_TABLE table=default.Table_fbceddd1104f4f24a7e21f0351add342 [id=fc194876b8324fba96d95217ea79d605]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:49.924763 714 consensus_peers.cc:487] T b8b64e4e64cc462891f1ee7a594d5026 P 289afc116aa74408aa5bff74e11c50d8 -> Peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Couldn't send request to peer f1e775296eb746db8b318f17d26820e6. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:52:49.924901 714 leader_election.cc:343] T d95f98ae7bdf42518bbff55e257fea88 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831): Not found: Tablet not found: d95f98ae7bdf42518bbff55e257fea88
I20250112 02:52:49.925029 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27752da73ed441629754b812e1fcf4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.924986 714 leader_election.cc:304] T d95f98ae7bdf42518bbff55e257fea88 P 289afc116aa74408aa5bff74e11c50d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 289afc116aa74408aa5bff74e11c50d8; no voters: f1e775296eb746db8b318f17d26820e6, fe36951a83fb4e57b26e187edbcbfe3d
I20250112 02:52:49.925220 3296 raft_consensus.cc:2743] T d95f98ae7bdf42518bbff55e257fea88 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:52:49.926393 1177 tablet_bootstrap.cc:492] T 54d3664071b146a3ac7c6885f9bda789 P 289afc116aa74408aa5bff74e11c50d8: No bootstrap required, opened a new log
I20250112 02:52:49.926499 1177 ts_tablet_manager.cc:1397] T 54d3664071b146a3ac7c6885f9bda789 P 289afc116aa74408aa5bff74e11c50d8: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.000s
I20250112 02:52:49.927080 1177 raft_consensus.cc:357] T 54d3664071b146a3ac7c6885f9bda789 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.927229 1177 raft_consensus.cc:383] T 54d3664071b146a3ac7c6885f9bda789 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:52:49.927284 1177 raft_consensus.cc:738] T 54d3664071b146a3ac7c6885f9bda789 P 289afc116aa74408aa5bff74e11c50d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289afc116aa74408aa5bff74e11c50d8, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.927440 1177 consensus_queue.cc:260] T 54d3664071b146a3ac7c6885f9bda789 P 289afc116aa74408aa5bff74e11c50d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.927439 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40610:
name: "default.Table_939b24fd526f41ba8bb3cb2103fb8a9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:52:49.927685 760 tablet_service.cc:1467] Processing CreateTablet for tablet 01b247f789f445ca9fb4b358c6f249ca (DEFAULT_TABLE table=default.Table_c3cfd78c4fd14e4ca682ba5ae32499d1 [id=f3ee16426dc14136947ec3e40e25d29d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:52:49.927914 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_939b24fd526f41ba8bb3cb2103fb8a9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:52:49.927994 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01b247f789f445ca9fb4b358c6f249ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.928002 1177 ts_tablet_manager.cc:1428] T 54d3664071b146a3ac7c6885f9bda789 P 289afc116aa74408aa5bff74e11c50d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250112 02:52:49.929342 1177 tablet_bootstrap.cc:492] T 2ecf730b6985404a8134b497a4b792c8 P 289afc116aa74408aa5bff74e11c50d8: Bootstrap starting.
I20250112 02:52:49.930675 1177 tablet_bootstrap.cc:654] T 2ecf730b6985404a8134b497a4b792c8 P 289afc116aa74408aa5bff74e11c50d8: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.931684 747 tablet_service.cc:1467] Processing CreateTablet for tablet 94c29d471dc347c6831e95abe4c943c4 (DEFAULT_TABLE table=default.Table_78a2b4e72b2f49aa95b57212aa2e418f [id=034f62124be7435ab14357aba3aeddc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.931953 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94c29d471dc347c6831e95abe4c943c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.933019 1043 raft_consensus.cc:1212] T 05000c6a48e0450d8e6685ca864f67f0 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.2] Dedup: 1.2->[]
I20250112 02:52:49.933887 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 436220c23fa948efb5b6ca15875d8b0e: default.Table_8fd4122d870a475baea2cc3cb97bd01a [id=7d048aa907a74605964b68726ac5d759] alter complete (version 1)
I20250112 02:52:49.934206 3292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42341,127.0.104.61:38771,127.0.104.60:39987
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:39987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=42831
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4135
ASAN enabled
I20250112 02:52:49.935674 753 tablet_service.cc:1467] Processing CreateTablet for tablet 6ff7a7cc250c446db0985ffc1bcc2f91 (DEFAULT_TABLE table=default.Table_17388940cd444538a523e34ad3cf51f1 [id=371bc05048924bd3a59e7a1057c07384]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.935910 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ff7a7cc250c446db0985ffc1bcc2f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.936357 3292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:52:49.814239 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 1db4f8b5532c4692968309fb803fb531 (DEFAULT_TABLE table=default.Table_ee4d7e840acc457c88cd3b4de3824bbe [id=632e835766a04e5bbb3d19c621434768]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.823405 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 878376c9593a4246a92de0a2fe42aca8 (DEFAULT_TABLE table=default.Table_5ad4c4df7d3944b898099c4681510214 [id=52ccea9492db4ed09ba48d53b0d0c0ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.937302 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 878376c9593a4246a92de0a2fe42aca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.940795 745 tablet_service.cc:1467] Processing CreateTablet for tablet a6b9a0fc1036472f968da8140b0e315c (DEFAULT_TABLE table=default.Table_72fbcfa736db454c930df858dc3b4baf [id=b07bcfaf824f442bb3b9597d79097e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.941066 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6b9a0fc1036472f968da8140b0e315c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.942770 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1db4f8b5532c4692968309fb803fb531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.944378 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc52271535df47e282bf1625c2e8e041" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6"
I20250112 02:52:49.944877 3292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 02:52:49.752228 1173 raft_consensus.cc:738] T fc52271535df47e282bf1625c2e8e041 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e775296eb746db8b318f17d26820e6, State: Initialized, Role: FOLLOWER
I20250112 02:52:49.831421 1042 raft_consensus.cc:1212] T 896908b2726c40f9aabdfc9a6b3283f7 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.3] Dedup: 1.3->[]
I20250112 02:52:49.835649 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc52271535df47e282bf1625c2e8e041" candidate_uuid: "289afc116aa74408aa5bff74e11c50d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:49.836462 3332 tablet.cc:1721] T 223eb8b4d1024985844867d5641600d1 P f1e775296eb746db8b318f17d26820e6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:52:49.854214 1020 tablet_service.cc:1467] Processing CreateTablet for tablet a6b9a0fc1036472f968da8140b0e315c (DEFAULT_TABLE table=default.Table_72fbcfa736db454c930df858dc3b4baf [id=b07bcfaf824f442bb3b9597d79097e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.947080 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6b9a0fc1036472f968da8140b0e315c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.947163 1173 consensus_queue.cc:260] T fc52271535df47e282bf1625c2e8e041 P f1e775296eb746db8b318f17d26820e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } }
I20250112 02:52:49.947690 1173 ts_tablet_manager.cc:1428] T fc52271535df47e282bf1625c2e8e041 P f1e775296eb746db8b318f17d26820e6: Time spent starting tablet: real 0.237s user 0.002s sys 0.000s
I20250112 02:52:49.947957 1173 tablet_bootstrap.cc:492] T 17c54c5dc86446cf85974777c8145ef6 P f1e775296eb746db8b318f17d26820e6: Bootstrap starting.
W20250112 02:52:49.822609 854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:44831, user_credentials={real_user=slave}} blocked reactor thread for 45950.6us
I20250112 02:52:49.823921 891 tablet_service.cc:1467] Processing CreateTablet for tablet 878376c9593a4246a92de0a2fe42aca8 (DEFAULT_TABLE table=default.Table_5ad4c4df7d3944b898099c4681510214 [id=52ccea9492db4ed09ba48d53b0d0c0ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.858872 885 tablet_service.cc:1467] Processing CreateTablet for tablet 27752da73ed441629754b812e1fcf4a9 (DEFAULT_TABLE table=default.Table_fbceddd1104f4f24a7e21f0351add342 [id=fc194876b8324fba96d95217ea79d605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.859028 897 tablet_service.cc:1467] Processing CreateTablet for tablet 94c29d471dc347c6831e95abe4c943c4 (DEFAULT_TABLE table=default.Table_78a2b4e72b2f49aa95b57212aa2e418f [id=034f62124be7435ab14357aba3aeddc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.948789 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 878376c9593a4246a92de0a2fe42aca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.948968 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27752da73ed441629754b812e1fcf4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.951587 3348 tablet.cc:1716] T 896908b2726c40f9aabdfc9a6b3283f7 P f1e775296eb746db8b318f17d26820e6: Skipping requested alter to schema version 4, tablet already version 4
I20250112 02:52:49.951647 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94c29d471dc347c6831e95abe4c943c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.855432 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 27752da73ed441629754b812e1fcf4a9 (DEFAULT_TABLE table=default.Table_fbceddd1104f4f24a7e21f0351add342 [id=fc194876b8324fba96d95217ea79d605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.952124 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27752da73ed441629754b812e1fcf4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.955252 3397 raft_consensus.cc:491] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:49.955353 3397 raft_consensus.cc:513] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:49.856685 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 01b247f789f445ca9fb4b358c6f249ca (DEFAULT_TABLE table=default.Table_c3cfd78c4fd14e4ca682ba5ae32499d1 [id=f3ee16426dc14136947ec3e40e25d29d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.955679 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01b247f789f445ca9fb4b358c6f249ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.955989 3397 leader_election.cc:290] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:49.859304 886 tablet_service.cc:1467] Processing CreateTablet for tablet 01b247f789f445ca9fb4b358c6f249ca (DEFAULT_TABLE table=default.Table_c3cfd78c4fd14e4ca682ba5ae32499d1 [id=f3ee16426dc14136947ec3e40e25d29d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.956321 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01b247f789f445ca9fb4b358c6f249ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.956594 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a243d008c17a46668a3d40e015e84f11" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289afc116aa74408aa5bff74e11c50d8" is_pre_election: true
W20250112 02:52:49.957264 855 leader_election.cc:343] T a243d008c17a46668a3d40e015e84f11 P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:52:49.959825 889 tablet_service.cc:1467] Processing CreateTablet for tablet a6b9a0fc1036472f968da8140b0e315c (DEFAULT_TABLE table=default.Table_72fbcfa736db454c930df858dc3b4baf [id=b07bcfaf824f442bb3b9597d79097e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.960315 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6b9a0fc1036472f968da8140b0e315c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.858261 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 94c29d471dc347c6831e95abe4c943c4 (DEFAULT_TABLE table=default.Table_78a2b4e72b2f49aa95b57212aa2e418f [id=034f62124be7435ab14357aba3aeddc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.858986 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 6ff7a7cc250c446db0985ffc1bcc2f91 (DEFAULT_TABLE table=default.Table_17388940cd444538a523e34ad3cf51f1 [id=371bc05048924bd3a59e7a1057c07384]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.962162 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94c29d471dc347c6831e95abe4c943c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.962332 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ff7a7cc250c446db0985ffc1bcc2f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.966997 1043 raft_consensus.cc:3054] T fc52271535df47e282bf1625c2e8e041 P f1e775296eb746db8b318f17d26820e6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:52:49.969215 1043 raft_consensus.cc:2463] T fc52271535df47e282bf1625c2e8e041 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289afc116aa74408aa5bff74e11c50d8 in term 1.
I20250112 02:52:49.972599 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f559dedfb7854cf890bef362c50b4213" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:49.972966 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a243d008c17a46668a3d40e015e84f11" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
W20250112 02:52:49.974963 989 consensus_peers.cc:487] T 83c04f1a2cb04950a9f6170e252c19b4 P f1e775296eb746db8b318f17d26820e6 -> Peer fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): Couldn't send request to peer fe36951a83fb4e57b26e187edbcbfe3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83c04f1a2cb04950a9f6170e252c19b4. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:52:49.975596 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 6d9e50a33ac74482aa9d4cdf9d61e91a (DEFAULT_TABLE table=default.Table_f7499edd95ff4f6d880e1712c2a1a19d [id=05b251913cda49fe9b12ec367ce4ad8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.975594 1021 tablet_service.cc:1467] Processing CreateTablet for tablet c30934b533b24eacad25a4a80eed4fd6 (DEFAULT_TABLE table=default.Table_e071703f861849acab78065ef7c67728 [id=629b42765c5a443b90a46b8663ef9843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.975955 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 268df1f410c4473ab4cb4c614a275045 (DEFAULT_TABLE table=default.Table_c1820d9fe8904d029ea65d852fcea59b [id=943f51db477a4e74ad732617a3012b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.975971 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d9e50a33ac74482aa9d4cdf9d61e91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.976231 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 268df1f410c4473ab4cb4c614a275045. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.976369 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c30934b533b24eacad25a4a80eed4fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.976682 891 tablet_service.cc:1467] Processing CreateTablet for tablet c30934b533b24eacad25a4a80eed4fd6 (DEFAULT_TABLE table=default.Table_e071703f861849acab78065ef7c67728 [id=629b42765c5a443b90a46b8663ef9843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.976961 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c30934b533b24eacad25a4a80eed4fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.981050 886 tablet_service.cc:1467] Processing CreateTablet for tablet 268df1f410c4473ab4cb4c614a275045 (DEFAULT_TABLE table=default.Table_c1820d9fe8904d029ea65d852fcea59b [id=943f51db477a4e74ad732617a3012b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.981993 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 268df1f410c4473ab4cb4c614a275045. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.981737 885 tablet_service.cc:1467] Processing CreateTablet for tablet 6d9e50a33ac74482aa9d4cdf9d61e91a (DEFAULT_TABLE table=default.Table_f7499edd95ff4f6d880e1712c2a1a19d [id=05b251913cda49fe9b12ec367ce4ad8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:52:49.982673 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d9e50a33ac74482aa9d4cdf9d61e91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:52:49.992655 1173 tablet_bootstrap.cc:654] T 17c54c5dc86446cf85974777c8145ef6 P f1e775296eb746db8b318f17d26820e6: Neither blocks nor log segments found. Creating new log.
I20250112 02:52:49.993312 3109 ts_tablet_manager.cc:1918] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:49.993450 3109 log.cc:1198] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-0/wal/wals/c29a9ca43e034b43a00207e5847bbee3
I20250112 02:52:49.993840 3109 ts_tablet_manager.cc:1939] T c29a9ca43e034b43a00207e5847bbee3 P 289afc116aa74408aa5bff74e11c50d8: Deleting consensus metadata
I20250112 02:52:49.994663 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 521) took 3205 ms. Trace:
I20250112 02:52:49.994793 3109 rpcz_store.cc:276] 0112 02:52:46.788863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.788993 (+ 130us) service_pool.cc:225] Handling call
0112 02:52:49.812127 (+3023134us) ts_tablet_manager.cc:1145] Deleting tablet c29a9ca43e034b43a00207e5847bbee3
0112 02:52:49.812136 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.846393 (+ 34257us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.993242 (+146849us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:49.994645 (+ 1403us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3021367}
I20250112 02:52:49.999751 3109 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49624 (request call id 517) took 3240 ms. Trace:
I20250112 02:52:49.999884 3109 rpcz_store.cc:276] 0112 02:52:46.759769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.762780 (+ 3011us) service_pool.cc:225] Handling call
0112 02:52:49.999641 (+3236861us) ts_tablet_manager.cc:1145] Deleting tablet 8fe57d374eb44c3fb1532484e214b654
0112 02:52:49.999652 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.999738 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3208686}
W20250112 02:52:50.000438 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet e1ac5b5ea3a6494baf806bd4441e747f (table default.Table_3f00edd22a8048609d508e9a9545a361 [id=b58c92930dda4f2c882eb730024971bb], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e1ac5b5ea3a6494baf806bd4441e747f consensus state has no leader
I20250112 02:52:50.002471 3397 raft_consensus.cc:491] T d1269a6ed80e46ecb7f4f87e5afb732f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:50.002583 3397 raft_consensus.cc:513] T d1269a6ed80e46ecb7f4f87e5afb732f P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44451 } }
I20250112 02:52:50.003074 3397 leader_election.cc:290] T d1269a6ed80e46ecb7f4f87e5afb732f P fe36951a83fb4e57b26e187edbcbfe3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289afc116aa74408aa5bff74e11c50d8 (127.0.104.1:35007), f1e775296eb746db8b318f17d26820e6 (127.0.104.3:44831)
I20250112 02:52:50.005710 3111 raft_consensus.cc:2267] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:50.006639 3111 ts_tablet_manager.cc:1905] T 781fe3514f4c4f80b1ee49278502ccc1 P f1e775296eb746db8b318f17d26820e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:50.008401 3110 ts_tablet_manager.cc:1918] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:52:50.008502 3110 log.cc:1198] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting WAL directory at /tmp/dist-test-tasko6SHYo/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650339756832-416-0/minicluster-data/ts-1/wal/wals/e477fc157aed4563923d85d66014293a
I20250112 02:52:50.008844 3110 ts_tablet_manager.cc:1939] T e477fc157aed4563923d85d66014293a P fe36951a83fb4e57b26e187edbcbfe3d: Deleting consensus metadata
I20250112 02:52:50.009689 3110 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38786 (request call id 400) took 3394 ms. Trace:
I20250112 02:52:50.009790 3110 rpcz_store.cc:276] 0112 02:52:46.615084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:52:46.615133 (+ 49us) service_pool.cc:225] Handling call
0112 02:52:49.779891 (+3164758us) ts_tablet_manager.cc:1145] Deleting tablet e477fc157aed4563923d85d66014293a
0112 02:52:49.779902 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:52:49.867199 (+ 87297us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:50.008323 (+141124us) tablet_metadata.cc:641] Metadata flushed
0112 02:52:50.009668 (+ 1345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3164591}
I20250112 02:52:50.010269 3110 tablet_replica.cc:331] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d: stopping tablet replica
I20250112 02:52:50.010233 440 catalog_manager.cc:4872] TS fe36951a83fb4e57b26e187edbcbfe3d (127.0.104.2:44451): tablet e477fc157aed4563923d85d66014293a (table default.Table_e495ee84abe3479c80c8ab2e14f86c96 [id=f5eb0a253e444d76978a22250c163a54]) successfully deleted
I20250112 02:52:50.010464 3110 raft_consensus.cc:2238] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:52:50.010607 3110 raft_consensus.cc:2267] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:52:50.011019 3110 ts_tablet_manager.cc:1905] T 8c12a1c5e3584450ac193743a0107e80 P fe36951a83fb4e57b26e187edbcbfe3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:52:50.011528 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1269a6ed80e46ecb7f4f87e5afb732f" candidate_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e775296eb746db8b318f17d26820e6" is_pre_election: true
I20250112 02:52:50.011930 3397 raft_consensus.cc:491] T 7eb24b3d8a9446e8abd9729462225002 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:52:50.012068 3397 raft_consensus.cc:513] T 7eb24b3d8a9446e8abd9729462225002 P fe36951a83fb4e57b26e187edbcbfe3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289afc116aa74408aa5bff74e11c50d8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35007 } } peers { permanent_uuid: "f1e775296eb746db8b318f17d26820e6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44831 } } peers { permanent_uuid: "fe36951a83fb4e57b26e187edbcbfe3d" member_type: VOTER last_known_addr { host: "127.0.104.2"